Mar 31 03:48:02 atlas-tds-mds1 kernel: [320941.219558] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 227s: evicting client at 82@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff88100a3d7000/0xe7f79ec3f66b62bf lrc: 3/0,0 mode: CR/CR res: [0x200003ae3:0x6ee:0x0].0 bits 0x9 rrc: 54 type: IBT flags: 0x200000000020 nid: 82@gni2 remote: 0x7870777e82c5dd04 expref: 10 pid: 12555 timeout: 4615487057 lvb_type: 0 used 0
Mar 31 03:48:02 atlas-tds-mds1 kernel: [320941.342532] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 4 previous similar messages
Mar 31 03:48:02 atlas-tds-mds1 kernel: [320941.372860] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 227s: evicting client at 6@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff881015153240/0xe7f79ec3f66b61d1 lrc: 3/0,0 mode: CR/CR res: [0x200003ae3:0x6ee:0x0].0 bits 0x9 rrc: 54 type: IBT flags: 0x200000000020 nid: 6@gni2 remote: 0x18a0370a108d6aa5 expref: 13 pid: 12468 timeout: 4615487057 lvb_type: 0 used 0
Mar 31 13:23:02 atlas-tds-mds1 kernel: [355453.871832] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 43@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff88101209ed80/0xe7f79ec3f6710314 lrc: 3/0,0 mode: CR/CR res: [0x200003ae3:0x717:0x0].0 bits 0x9 rrc: 52 type: IBT flags: 0x200000000020 nid: 43@gni2 remote: 0x4cb9685ef5e4e6a8 expref: 12 pid: 12586 timeout: 4649987437 lvb_type: 0 used 0
Mar 31 13:23:02 atlas-tds-mds1 kernel: [355453.989053] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message
Mar 31 13:23:02 atlas-tds-mds1 kernel: [355454.222842] LustreError: 11630:0:(ldlm_lockd.c:2380:ldlm_cancel_handler()) ldlm_cancel from 43@gni2 arrived at 1396286582 with bad export cookie 16715003106496974761
Mar 31 14:09:51 atlas-tds-mds1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Initializing cgroup subsys cpuset
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Initializing cgroup subsys cpu
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Linux version 2.6.32-358.23.2.el6.atlas.x86_64 (jsimmons@lgmgmt1) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Mon Mar 31 10:15:50 EDT 2014
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Command line: initrd=initrd-2.6.32-358.23.2.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 crashkernel=192M BOOT_IMAGE=vmlinuz-2.6.32-358.23.2.el6.atlas.x86_64 BOOTIF=01-f0-4d-a2-73-cc-ed 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] KERNEL supported cpus:
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Intel GenuineIntel
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   AMD AuthenticAMD
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Centaur CentaurHauls
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] BIOS-provided physical RAM map:
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 0000000000100000 - 00000000783d6000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000783d6000 - 00000000797d6000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000797d6000 - 000000007e2ff000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000007e2ff000 - 000000007eaff000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000007eaff000 - 000000007eeff000 (ACPI NVS)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000007eeff000 - 000000007efff000 (ACPI data)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000007efff000 - 000000007f000000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 000000007f000000 - 0000000090000000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000feb00000 - 00000000feb04000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000fed18000 - 00000000fed19000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  BIOS-e820: 0000000100000000 - 0000001080000000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] DMI 2.7 present.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SMBIOS version 2.7 @ 0xFE120
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] DMI: Dell Inc. PowerEdge C6220 II/09N44V, BIOS 2.2.3 11/07/2013
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] last_pfn = 0x1080000 max_arch_pfn = 0x400000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] MTRR default type: uncachable
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] MTRR fixed ranges enabled:
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   00000-9FFFF write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   A0000-BFFFF uncachable
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   C0000-EFFFF write-protect
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   F0000-FFFFF write-combining
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] MTRR variable ranges enabled:
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   0 base 000000000000 mask 3FFFC0000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   1 base 0000FFC00000 mask 3FFFFFC00000 write-protect
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   2 base 000040000000 mask 3FFFC0000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   3 disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   4 base 000100000000 mask 3FFF00000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   5 base 000200000000 mask 3FFE00000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   6 base 000400000000 mask 3FFC00000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   7 base 000800000000 mask 3FF800000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   8 base 001000000000 mask 3FFF80000000 write-back
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   9 disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] last_pfn = 0x7f000 max_arch_pfn = 0x400000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] initial memory mapped : 0 - 20000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Using GB pages for direct mapping
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] init_memory_mapping: 0000000000000000-000000007f000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  0000000000 - 0040000000 page 1G
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  0040000000 - 007f000000 page 2M
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] kernel direct mapping tables up to 7f000000 @ 8000-a000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] init_memory_mapping: 0000000100000000-0000001080000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  0100000000 - 1080000000 page 1G
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] kernel direct mapping tables up to 1080000000 @ 9000-a000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] RAMDISK: 7767a000 - 783b54e6
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: RSDP 00000000000fe020 00024 (v02 DELL  )
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: XSDT 000000007effe170 000A4 (v01 DELL   DCSRADON 00000001      01000013)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: FACP 000000007effa000 0010C (v05 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI Warning: FADT (revision 5) is longer than ACPI 2.0 version, truncating length 0x10C to 0xF4 (20090903/tbfadt-288)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: DSDT 000000007efea000 0BDBD (v01 DELL   DCSRADON 00000000 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: FACS 000000007eda0000 00040
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: UEFI 000000007effd000 00236 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SPMI 000000007effc000 00040 (v05 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: BOOT 000000007effb000 00028 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: APIC 000000007eff8000 00436 (v03 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: MCFG 000000007eff7000 0003C (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SLIC 000000007eff6000 00176 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SRAT 000000007efe8000 001A8 (v02 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SLIT 000000007efe7000 0003C (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: BDAT 000000007efe6000 00030 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: PRAD 000000007efe5000 000BC (v02 PRADID  PRADTID 00000001 ACPI 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: FPDT 000000007efe3000 00044 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SSDT 000000007ef10000 D2E70 (v02 DELL   DCSRADON 00004000 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: HPET 000000007eff9000 00038 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: WDAT 000000007ef0f000 00194 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: SPCR 000000007efe9000 00050 (v01 DELL   DCSRADON 00000001 DELL 00040000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Setting APIC routing to flat.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 2 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 4 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 6 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 8 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 10 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 12 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 0 -> APIC 14 -> Node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 32 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 34 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 36 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 38 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 40 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 42 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 44 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: PXM 1 -> APIC 46 -> Node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: Node 0 PXM 0 0-80000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: Node 0 PXM 0 100000000-880000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SRAT: Node 1 PXM 1 880000000-1080000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] NUMA: Using 31 for the hash shift.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Bootmem setup node 0 0000000000000000-0000000880000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   NODE_DATA [0000000000009000 - 000000000003cfff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   bootmap [0000000000100000 -  000000000020ffff] pages 110
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0880000000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #2 [0001000000 - 000201b0e4]    TEXT DATA BSS ==> [0001000000 - 000201b0e4]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #3 [007767a000 - 00783b54e6]          RAMDISK ==> [007767a000 - 00783b54e6]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #4 [000009cc00 - 0000100000]    BIOS reserved ==> [000009cc00 - 0000100000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #5 [000201c000 - 000201c31f]              BRK ==> [000201c000 - 000201c31f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #7 [0000001000 - 000000103c]        ACPI SLIT ==> [0000001000 - 000000103c]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Bootmem setup node 1 0000000880000000-0000001080000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   NODE_DATA [0000000880000040 - 000000088003403f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   bootmap [0000000880035000 -  0000000880134fff] pages 100
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] (8 early reservations) ==> bootmem [0880000000 - 1080000000]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #2 [0001000000 - 000201b0e4]    TEXT DATA BSS
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #3 [007767a000 - 00783b54e6]          RAMDISK
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #4 [000009cc00 - 0000100000]    BIOS reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #5 [000201c000 - 000201c31f]              BRK
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #6 [0000008000 - 0000009000]          PGTABLE
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   #7 [0000001000 - 000000103c]        ACPI SLIT
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] found SMP MP-table at [ffff8800000fe1b0] fe1b0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Reserving 192MB of memory at 48MB for crashkernel (System RAM: 67584MB)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  [ffffea0000000000-ffffea001dbfffff] PMD -> [ffff880028600000-ffff8800445fffff] on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]  [ffffea001dc00000-ffffea0039bfffff] PMD -> [ffff880880200000-ffff88089c1fffff] on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Zone PFN ranges:
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA      0x00000001 -> 0x00001000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA32    0x00001000 -> 0x00100000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Normal   0x00100000 -> 0x01080000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Movable zone start PFN for each node
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] early_node_map[6] active PFN ranges
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     0: 0x00000001 -> 0x0000009d
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     0: 0x00000100 -> 0x000783d6
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     0: 0x000797d6 -> 0x0007e2ff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     0: 0x0007efff -> 0x0007f000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     0: 0x00100000 -> 0x00880000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]     1: 0x00880000 -> 0x01080000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] On node 0 totalpages: 8375964
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA zone: 56 pages used for memmap
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA zone: 103 pages reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA zone: 3837 pages, LIFO batch:0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA32 zone: 14280 pages used for memmap
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   DMA32 zone: 493368 pages, LIFO batch:31
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Normal zone: 107520 pages used for memmap
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Normal zone: 7756800 pages, LIFO batch:31
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] On node 1 totalpages: 8388608
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Normal zone: 114688 pages used for memmap
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000]   Normal zone: 8273920 pages, LIFO batch:31
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: PM-Timer IO Port: 0x408
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Setting APIC routing to flat.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x4f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x58] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x59] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x5f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6a] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6b] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6c] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6d] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6e] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x6f] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0xff] disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: X2APIC_NMI (uid[0xffffffff] high edge lint[0x1])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IRQ0 used by override.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IRQ2 used by override.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: IRQ9 used by override.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Using ACPI (MADT) for SMP configuration information
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] SMP: Allowing 120 CPUs, 104 hotplug CPUs
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] nr_irqs_gsi: 72
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 000000000009d000 - 00000000000a0000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000783d6000 - 00000000797d6000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 000000007e2ff000 - 000000007eaff000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 000000007eaff000 - 000000007eeff000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 000000007eeff000 - 000000007efff000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 000000007f000000 - 0000000090000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 0000000090000000 - 00000000feb00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000feb00000 - 00000000feb04000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000feb04000 - 00000000fec00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fec01000 - 00000000fed18000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fed18000 - 00000000fed19000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fed19000 - 00000000fed1c000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fed1c000 - 00000000fed90000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fed90000 - 00000000fee00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000ffc00000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PM: Registered nosave memory: 00000000ffc00000 - 0000000100000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Allocating PCI resources starting at 90000000 (gap: 90000000:6eb00000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Booting paravirtualized kernel on bare hardware
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] NR_CPUS:4096 nr_cpumask_bits:120 nr_cpu_ids:120 nr_node_ids:2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PERCPU: Embedded 31 pages/cpu @ffff880044600000 s94616 r8192 d24168 u131072
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 016 018 020 022 024 026 028 030 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 --- --- --- --- 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [1] 008 009 010 011 012 013 014 015 017 019 021 023 025 027 029 031 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 --- --- --- --- 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16527925
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Policy zone: Normal
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Kernel command line: initrd=initrd-2.6.32-358.23.2.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 crashkernel=192M BOOT_IMAGE=vmlinuz-2.6.32-358.23.2.el6.atlas.x86_64 BOOTIF=01-f0-4d-a2-73-cc-ed 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] audit: disabled (until reboot)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Checking aperture...
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] No AGP bridge found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] software IO TLB at phys 0x20000000 - 0x24000000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Memory: 65832304k/69206016k available (5216k kernel code, 2147728k absent, 1225984k reserved, 7126k data, 1264k init)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Hierarchical RCU implementation.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] NR_IRQS:33024 nr_irqs:2184
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Extended CMOS year: 2000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Console: colour VGA+ 80x25
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] console [tty0] enabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] console [ttyS1] enabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] allocated 268435456 bytes of page_cgroup
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] hpet clockevent registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000000] Fast TSC calibration using PIT
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.001000] Detected 2600.129 MHz processor.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.000038] Calibrating delay loop (skipped), value calculated using timer frequency.. 5200.25 BogoMIPS (lpj=2600129)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.030084] pid_max: default: 122880 minimum: 960
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.050080] Security Framework initialized
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.060047] SELinux:  Disabled at boot.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.075689] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.113121] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.135274] Mount-cache hash table entries: 256
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.140335] Initializing cgroup subsys ns
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.150290] Initializing cgroup subsys cpuacct
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.169928] Initializing cgroup subsys memory
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.180204] Initializing cgroup subsys devices
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.199743] Initializing cgroup subsys freezer
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.209942] Initializing cgroup subsys net_cls
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.220356] Initializing cgroup subsys blkio
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.239941] Initializing cgroup subsys perf_event
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.250285] Initializing cgroup subsys net_prio
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.269779] CPU: Physical Processor ID: 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.279913] CPU: Processor Core ID: 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.290909] mce: CPU supports 25 MCE banks
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.300194] CPU0: Thermal monitoring enabled (TM1)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.321280] using mwait in idle threads.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.331055] ACPI: Core revision 20090903
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.391144] ftrace: converting mcount calls to 0f 1f 44 00 00
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.400516] ftrace: allocating 21440 entries in 85 pages
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.427788] x2apic not enabled, IRQ remapping init failed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.440412] APIC routing finalized to physical flat.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.451900] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.481330] CPU0: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz stepping 04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.602075] Performance Events: PEBS fmt1+, generic architected perfmon, Intel PMU driver.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.621943] ... version:                3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.640118] ... bit width:              48
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.650710] ... generic registers:      8
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.661250] ... value mask:             0000ffffffffffff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.681073] ... max period:             000000007fffffff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.700735] ... fixed-purpose events:   3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.711113] ... event mask:             00000007000000ff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.732717] NMI watchdog enabled, takes one hw-pmu counter.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.750690] Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 Ok.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    0.991113] Booting Node   1, Processors  #8 #9 #10 #11 #12 #13 #14 #15
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.320164] Brought up 16 CPUs
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.330347] Total of 16 processors activated (83196.01 BogoMIPS).
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364952] sizeof(vma)=200 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364954] sizeof(page)=56 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364955] sizeof(inode)=592 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364956] sizeof(dentry)=192 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364957] sizeof(ext3inode)=800 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364958] sizeof(buffer_head)=104 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364958] sizeof(skbuff)=232 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.364959] sizeof(task_struct)=2648 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.365790] devtmpfs: initialized
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.376973] PM: Registering ACPI NVS region at 7eaff000 (4194304 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.391090] regulator: core version 0.5
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.401358] NET: Registered protocol family 16
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.421318] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.441603] ACPI: bus type pci registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.451809] PCI: MCFG configuration 0: base 80000000 segment 0 buses 0 - 255
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.480803] PCI: MCFG area at 80000000 reserved in E820
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.528066] PCI: Using MMCONFIG at 80000000 - 8fffffff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.531543] PCI: Using configuration type 1 for base access
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.553708] bio: create slab <bio-0> at 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.575188] ACPI: EC: Look up EC in DSDT
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.810779] ACPI: Dynamic OEM Table Load - [PRAD] OemId [PRADID] OemTableId [PRADTID]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.832580] ACPI: Interpreter enabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.841843] ACPI: (supports S0 S4 S5)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.860571] ACPI: Using IOAPIC for interrupt routing
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.880266] ACPI: No dock devices found.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.890950] HEST: Table not found.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.901124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.932226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.952135] pci_root PNP0A08:00: host bridge window [io  0x0000-0x0cf7]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.971581] pci_root PNP0A08:00: host bridge window [io  0x1000-0xdfff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    1.992044] pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.021425] pci_root PNP0A08:00: host bridge window [mem 0x90000000-0xebffefff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.042033] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.061610] pci 0000:00:00.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.080691] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.100646] pci 0000:00:01.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.111482] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.131633] pci 0000:00:02.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.150678] pci 0000:00:04.0: reg 10: [mem 0x91300000-0x91303fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.150754] pci 0000:00:04.1: reg 10: [mem 0x91304000-0x91307fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.150829] pci 0000:00:04.2: reg 10: [mem 0x91308000-0x9130bfff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.150904] pci 0000:00:04.3: reg 10: [mem 0x9130c000-0x9130ffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.150982] pci 0000:00:04.4: reg 10: [mem 0x91310000-0x91313fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.151057] pci 0000:00:04.5: reg 10: [mem 0x91314000-0x91317fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.151132] pci 0000:00:04.6: reg 10: [mem 0x91318000-0x9131bfff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.151207] pci 0000:00:04.7: reg 10: [mem 0x9131c000-0x9131ffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.151472] pci 0000:00:05.4: reg 10: [mem 0x91327000-0x91327fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.151599] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.162178] pci 0000:00:11.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.181316] pci 0000:00:16.0: reg 10: [mem 0x91320000-0x9132000f 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.181367] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.201448] pci 0000:00:16.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.212023] pci 0000:00:16.1: reg 10: [mem 0x91321000-0x9132100f 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.212073] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.231994] pci 0000:00:16.1: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.251374] pci 0000:00:1a.0: reg 10: [mem 0x91326000-0x913263ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.254154] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.271354] pci 0000:00:1a.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.282128] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.301836] pci 0000:00:1c.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.321187] pci 0000:00:1d.0: reg 10: [mem 0x91325000-0x913253ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.323962] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.340728] pci 0000:00:1d.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351511] pci 0000:00:1f.2: reg 10: [io  0x4048-0x404f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351518] pci 0000:00:1f.2: reg 14: [io  0x4054-0x4057]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351525] pci 0000:00:1f.2: reg 18: [io  0x4040-0x4047]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351532] pci 0000:00:1f.2: reg 1c: [io  0x4050-0x4053]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351539] pci 0000:00:1f.2: reg 20: [io  0x4020-0x403f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351546] pci 0000:00:1f.2: reg 24: [mem 0x91324000-0x913247ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.351576] pci 0000:00:1f.2: PME# supported from D3hot
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.370985] pci 0000:00:1f.2: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381729] pci 0000:00:1f.3: reg 10: [mem 0x91322000-0x913220ff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381748] pci 0000:00:1f.3: reg 20: [io  0x4000-0x401f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381798] pci 0000:02:00.0: reg 10: [mem 0x91220000-0x9123ffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381811] pci 0000:02:00.0: reg 18: [io  0x3020-0x303f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381818] pci 0000:02:00.0: reg 1c: [mem 0x91244000-0x91247fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.381856] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.401529] pci 0000:02:00.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412409] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412424] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412458] pci 0000:02:00.3: reg 10: [mem 0x91200000-0x9121ffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412471] pci 0000:02:00.3: reg 18: [io  0x3000-0x301f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412477] pci 0000:02:00.3: reg 1c: [mem 0x91240000-0x91243fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.412519] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.432115] pci 0000:02:00.3: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.451292] pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.451307] pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.451381] pci 0000:00:01.0: PCI bridge to [bus 02-02]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470822] pci 0000:00:01.0:   bridge window [io  0x3000-0x3fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470825] pci 0000:00:01.0:   bridge window [mem 0x91200000-0x912fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470830] pci 0000:00:01.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470849] pci 0000:03:00.0: reg 10: [io  0x2000-0x20ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470856] pci 0000:03:00.0: reg 14: [mem 0x91140000-0x91143fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470864] pci 0000:03:00.0: reg 1c: [mem 0x91100000-0x9113ffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470874] pci 0000:03:00.0: reg 30: [mem 0xfffe0000-0xffffffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470890] pci 0000:03:00.0: supports D1 D2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.470937] pci 0000:00:02.0: PCI bridge to [bus 03-03]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.481836] pci 0000:00:02.0:   bridge window [io  0x2000-0x2fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.481839] pci 0000:00:02.0:   bridge window [mem 0x91100000-0x911fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.481844] pci 0000:00:02.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.481882] pci 0000:00:11.0: PCI bridge to [bus 04-04]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.501418] pci 0000:00:11.0:   bridge window [io  0xf000-0x0000] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.501422] pci 0000:00:11.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.501428] pci 0000:00:11.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.501552] pci 0000:05:00.0: supports D1 D2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.501554] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.521797] pci 0000:05:00.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.541045] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552237] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552240] pci 0000:00:1c.0:   bridge window [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552245] pci 0000:00:1c.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552301] pci 0000:06:00.0: reg 10: [mem 0x90000000-0x90ffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552319] pci 0000:06:00.0: reg 14: [mem 0x91000000-0x9101ffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552337] pci 0000:06:00.0: reg 18: [io  0x1000-0x107f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552440] pci 0000:06:00.0: supports D1 D2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.552441] pci 0000:06:00.0: PME# supported from D0 D1 D2 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.580869] pci 0000:06:00.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.591507] pci 0000:05:00.0: PCI bridge to [bus 06-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.611061] pci 0000:05:00.0:   bridge window [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.611066] pci 0000:05:00.0:   bridge window [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.611075] pci 0000:05:00.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.611127] pci 0000:00:1e.0: PCI bridge to [bus 07-07] (subtractive decode)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631635] pci 0000:00:1e.0:   bridge window [io  0xf000-0x0000] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631638] pci 0000:00:1e.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631643] pci 0000:00:1e.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631645] pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631646] pci 0000:00:1e.0:   bridge window [io  0x1000-0xdfff] (subtractive decode)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631648] pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631649] pci 0000:00:1e.0:   bridge window [mem 0x90000000-0xebffefff] (subtractive decode)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631671] pci_bus 0000:00: on NUMA node 0 (pxm 0)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631674] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631853] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE01._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631898] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE03._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.631962] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EVRP._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.632005] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.RP01._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.632059] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.632276]  pci0000:00: Requesting ACPI _OSC control (0x1d)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.651676]  pci0000:00: ACPI _OSC control (0x1d) granted
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.678568] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.692286]  pci0000:7f: Requesting ACPI _OSC control (0x1d)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.702422]  pci0000:7f: ACPI _OSC control (0x1d) granted
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.723134] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.742279] pci_root PNP0A08:01: host bridge window [io  0x0000-0x0cf7]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.762345] pci_root PNP0A08:01: host bridge window [io  0xe000-0xffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.790963] pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.812035] pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.832804] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.860949] pci 0000:80:02.0: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.871740] pci 0000:80:02.2: PME# supported from D0 D3hot D3cold
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.891702] pci 0000:80:02.2: PME# disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902575] pci 0000:80:04.0: reg 10: [mem 0xec900000-0xec903fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902657] pci 0000:80:04.1: reg 10: [mem 0xec904000-0xec907fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902737] pci 0000:80:04.2: reg 10: [mem 0xec908000-0xec90bfff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902816] pci 0000:80:04.3: reg 10: [mem 0xec90c000-0xec90ffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902896] pci 0000:80:04.4: reg 10: [mem 0xec910000-0xec913fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.902975] pci 0000:80:04.5: reg 10: [mem 0xec914000-0xec917fff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.903055] pci 0000:80:04.6: reg 10: [mem 0xec918000-0xec91bfff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.903134] pci 0000:80:04.7: reg 10: [mem 0xec91c000-0xec91ffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.903413] pci 0000:80:05.4: reg 10: [mem 0xec920000-0xec920fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.903504] pci 0000:80:02.0: PCI bridge to [bus 81-81]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.922146] pci 0000:80:02.0:   bridge window [io  0xf000-0x0000] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.922149] pci 0000:80:02.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.922153] pci 0000:80:02.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.922541] pci 0000:82:00.0: reg 10: [mem 0xec800000-0xec8fffff 64bit]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.922721] pci 0000:82:00.0: reg 18: [mem 0xec000000-0xec7fffff 64bit pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.923066] pci 0000:82:00.0: reg 30: [mem 0xfff00000-0xffffffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.923938] pci 0000:80:02.2: PCI bridge to [bus 82-82]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941753] pci 0000:80:02.2:   bridge window [io  0xf000-0x0000] (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941756] pci 0000:80:02.2:   bridge window [mem 0xec800000-0xec8fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941761] pci 0000:80:02.2:   bridge window [mem 0xec000000-0xec7fffff 64bit pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941771] pci_bus 0000:80: on NUMA node 1 (pxm 1)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941773] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941888] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PE13._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.941933] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PE15._PRT]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.942044]  pci0000:80: Requesting ACPI _OSC control (0x1d)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.961525]  pci0000:80: ACPI _OSC control (0x1d) granted
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    2.982965] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.002478]  pci0000:ff: Requesting ACPI _OSC control (0x1d)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.021115]  pci0000:ff: ACPI _OSC control (0x1d) granted
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.033140] ACPI: PCI Interrupt Link [LNKA] (IRQs 1 3 *4 5 6 10 11 12 14 15)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.061854] ACPI: PCI Interrupt Link [LNKB] (IRQs 1 3 4 *5 6 10 11 12 14 15)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.082813] ACPI: PCI Interrupt Link [LNKC] (IRQs 1 3 4 5 *6 10 11 12 14 15)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.104038] ACPI: PCI Interrupt Link [LNKD] (IRQs 1 3 4 5 6 10 11 12 14 15) *9
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.132482] ACPI: PCI Interrupt Link [LNKE] (IRQs 1 3 4 5 6 10 11 12 14 15) *0, disabled.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.154172] ACPI: PCI Interrupt Link [LNKF] (IRQs 1 3 4 5 6 10 *11 12 14 15)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.182151] ACPI: PCI Interrupt Link [LNKG] (IRQs 1 3 4 5 6 10 11 12 14 15) *0, disabled.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.211393] ACPI: PCI Interrupt Link [LNKH] (IRQs 1 3 4 5 6 *10 11 12 14 15)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.232456] vgaarb: device added: PCI:0000:06:00.0,decodes=io+mem,owns=io+mem,locks=none
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.261683] vgaarb: loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.271103] vgaarb: bridge control possible 0000:06:00.0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.282523] SCSI subsystem initialized
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.301443] libata version 3.00 loaded.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.301536] usbcore: registered new interface driver usbfs
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.312500] usbcore: registered new interface driver hub
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.331968] usbcore: registered new device driver usb
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.352093] PCI: Using ACPI for IRQ routing
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.362139] PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.362140] PCI: pci_cache_line_size set to 64 bytes
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.362555] NetLabel: Initializing
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.372254] NetLabel:  domain hash size = 128
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.391313] NetLabel:  protocols = UNLABELED CIPSOv4
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.402271] NetLabel:  unlabeled traffic allowed by default
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422196]   alloc irq_desc for 72 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422197]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422204]   alloc irq_desc for 73 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422205]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422209]   alloc irq_desc for 74 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422210]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422215]   alloc irq_desc for 75 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422216]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422220]   alloc irq_desc for 76 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422221]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.422224] HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.442600] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 72, 73, 74, 75, 76, 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.471755] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.493759] hpet: hpet2 irq 72 for MSI
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.494447] hpet: hpet3 irq 73 for MSI
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.495039] hpet: hpet4 irq 74 for MSI
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.495623] hpet: hpet5 irq 75 for MSI
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.496220] hpet: hpet6 irq 76 for MSI
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.501694] Switching to clocksource hpet
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.520638] pnp: PnP ACPI init
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.522878] ACPI: bus type pnp registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542000] pnp 00:00: [io  0x0cf8-0x0cff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542094] pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542245] pnp 00:01: [mem 0xfed1c000-0xfed1ffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542247] pnp 00:01: [mem 0xebfff000-0xebffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542249] pnp 00:01: [mem 0xfed20000-0xfed3ffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542250] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542252] pnp 00:01: [mem 0xff000000-0xffffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542254] pnp 00:01: [mem 0xfee00000-0xfeefffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542381] pnp 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542565] pnp 00:02: [io  0x0ca8]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542567] pnp 00:02: [io  0x0cac]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542592] pnp 00:02: Plug and Play ACPI device, IDs IPI0001 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542601] pnp 00:03: [io  0x0000-0x001f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542603] pnp 00:03: [io  0x0081-0x0091]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542604] pnp 00:03: [io  0x0093-0x009f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542605] pnp 00:03: [io  0x00c0-0x00df]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542607] pnp 00:03: [dma 4]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542631] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542638] pnp 00:04: [mem 0xff000000-0xffffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542661] pnp 00:04: Plug and Play ACPI device, IDs INT0800 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542742] pnp 00:05: [mem 0xfed00000-0xfed003ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542767] pnp 00:05: Plug and Play ACPI device, IDs PNP0103 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542777] pnp 00:06: [io  0x00f0]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542786] pnp 00:06: [irq 13]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542809] pnp 00:06: Plug and Play ACPI device, IDs PNP0c04 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542825] pnp 00:07: [io  0x002e-0x002f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542826] pnp 00:07: [io  0x004e-0x004f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542828] pnp 00:07: [io  0x0061]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542829] pnp 00:07: [io  0x0063]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542830] pnp 00:07: [io  0x0065]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542831] pnp 00:07: [io  0x0067]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542833] pnp 00:07: [io  0x0070]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542834] pnp 00:07: [io  0x0080]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542835] pnp 00:07: [io  0x0092]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542837] pnp 00:07: [io  0x00b2-0x00b3]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542838] pnp 00:07: [io  0x0680-0x069f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542839] pnp 00:07: [io  0x0400-0x0453]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542841] pnp 00:07: [io  0x0458-0x047f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542842] pnp 00:07: [io  0x0500-0x057f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542843] pnp 00:07: [io  0x164e-0x164f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.542858] pnp 00:07: disabling [io  0x164e-0x164f] because it overlaps 0000:00:1c.0 BAR 13 [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.572742] pnp 00:07: disabling [io  0x164e-0x164f disabled] because it overlaps 0000:05:00.0 BAR 13 [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612104] pnp 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612112] pnp 00:08: [io  0x0070-0x0077]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612117] pnp 00:08: [irq 8]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612143] pnp 00:08: Plug and Play ACPI device, IDs PNP0b00 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612166] pnp 00:09: [io  0x0454-0x0457]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612253] pnp 00:09: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612530] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (disabled)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612649] pnp 00:0b: Plug and Play ACPI device, IDs PNP0a03 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612816] pnp 00:0c: [io  0x0cf8-0x0cff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612859] pnp 00:0c: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.612993] pnp 00:0d: Plug and Play ACPI device, IDs PNP0a03 (active)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.613442] pnp: PnP ACPI: found 14 devices
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.622466] ACPI: ACPI bus type pnp unregistered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.641432] system 00:01: [mem 0xfed1c000-0xfed1ffff] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.661823] system 00:01: [mem 0xebfff000-0xebffffff] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.682081] system 00:01: [mem 0xfed20000-0xfed3ffff] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.702365] system 00:01: [mem 0xff000000-0xffffffff] could not be reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.731300] system 00:01: [mem 0xfee00000-0xfeefffff] could not be reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.751875] system 00:07: [io  0x0680-0x069f] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.771903] system 00:07: [io  0x0400-0x0453] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.791842] system 00:07: [io  0x0458-0x047f] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.811889] system 00:07: [io  0x0500-0x057f] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.831738] system 00:09: [io  0x0454-0x0457] has been reserved
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.856437] pci 0000:03:00.0: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.881969] pci 0000:82:00.0: no compatible bridge window for [mem 0xfff00000-0xffffffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.911774] PCI: max bus depth: 2 pci_try_num: 3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.911841] pci 0000:00:02.0: BAR 15: assigned [mem 0x91400000-0x914fffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932495] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932501] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932508] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932514] pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932520] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932527] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932533] pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932539] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932546] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932552] pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932558] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932565] pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.932568] pci 0000:02:00.0: BAR 7: assigned [mem 0x91248000-0x91267fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.952540] pci 0000:02:00.0: BAR 7: set to [mem 0x91248000-0x91267fff] (PCI address [0x91248000-0x91267fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.983271] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.983277] pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    3.983280] pci 0000:02:00.0: BAR 10: assigned [mem 0x91268000-0x91287fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.011769] pci 0000:02:00.0: BAR 10: set to [mem 0x91268000-0x91287fff] (PCI address [0x91268000-0x91287fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.042513] pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.042519] pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.042522] pci 0000:02:00.3: BAR 7: assigned [mem 0x91288000-0x912a7fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.062511] pci 0000:02:00.3: BAR 7: set to [mem 0x91288000-0x912a7fff] (PCI address [0x91288000-0x912a7fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.093372] pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.093379] pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.093381] pci 0000:02:00.3: BAR 10: assigned [mem 0x912a8000-0x912c7fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.121854] pci 0000:02:00.3: BAR 10: set to [mem 0x912a8000-0x912c7fff] (PCI address [0x912a8000-0x912c7fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.152760] pci 0000:00:01.0: PCI bridge to [bus 02-02]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.172039] pci 0000:00:01.0: PCI bridge to [bus 02-02]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.191676] pci 0000:00:01.0:   bridge window [io  0x3000-0x3fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.211721] pci 0000:00:01.0:   bridge window [mem 0x91200000-0x912fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.232056] pci 0000:00:01.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.252239] pci 0000:03:00.0: BAR 6: assigned [mem 0x91400000-0x9141ffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.273328] pci 0000:00:02.0: PCI bridge to [bus 03-03]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.292614] pci 0000:00:02.0: PCI bridge to [bus 03-03]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.322724] pci 0000:00:02.0:   bridge window [io  0x2000-0x2fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.341551] pci 0000:00:02.0:   bridge window [mem 0x91100000-0x911fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.362120] pci 0000:00:02.0:   bridge window [mem 0x91400000-0x914fffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.383209] pci 0000:00:11.0: PCI bridge to [bus 04-04]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.402439] pci 0000:00:11.0: PCI bridge to [bus 04-04]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.422345] pci 0000:00:11.0:   bridge window [io  disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.442134] pci 0000:00:11.0:   bridge window [mem disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.461772] pci 0000:00:11.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.481981] pci 0000:05:00.0: PCI bridge to [bus 06-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.493024] pci 0000:05:00.0: PCI bridge to [bus 06-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.512563] pci 0000:05:00.0:   bridge window [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.532735] pci 0000:05:00.0:   bridge window [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.552875] pci 0000:05:00.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.573036] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.592556] pci 0000:00:1c.0: PCI bridge to [bus 05-06]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.612009] pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.632224] pci 0000:00:1c.0:   bridge window [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.652571] pci 0000:00:1c.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.672765] pci 0000:00:1e.0: PCI bridge to [bus 07-07]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.692231] pci 0000:00:1e.0: PCI bridge to [bus 07-07]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.711826] pci 0000:00:1e.0:   bridge window [io  disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.723270] pci 0000:00:1e.0:   bridge window [mem disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.743093] pci 0000:00:1e.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.763069] pci 0000:80:02.0: PCI bridge to [bus 81-81]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.782630] pci 0000:80:02.0: PCI bridge to [bus 81-81]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.802113] pci 0000:80:02.0:   bridge window [io  disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.822080] pci 0000:80:02.0:   bridge window [mem disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.841887] pci 0000:80:02.0:   bridge window [mem pref disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.861905] pci 0000:82:00.0: BAR 6: can't assign mem pref (size 0x100000)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.882359] pci 0000:80:02.2: PCI bridge to [bus 82-82]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.901872] pci 0000:80:02.2: PCI bridge to [bus 82-82]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.912988] pci 0000:80:02.2:   bridge window [io  disabled]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.932992] pci 0000:80:02.2:   bridge window [mem 0xec800000-0xec8fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.953275] pci 0000:80:02.2:   bridge window [mem 0xec000000-0xec7fffff 64bit pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.983125]   alloc irq_desc for 26 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.983126]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    4.983134] pci 0000:00:01.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.002865] pci 0000:00:01.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.002886]   alloc irq_desc for 32 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.002887]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.002892] pci 0000:00:02.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.031835] pci 0000:00:02.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.031853]   alloc irq_desc for 16 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.031854]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.031861] pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.052009] pci 0000:00:11.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.052024]   alloc irq_desc for 17 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.052025]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.052029] pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.072540] pci 0000:00:1c.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.072571] pci 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.092956] pci 0000:05:00.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.092963] pci 0000:00:1e.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.092983]   alloc irq_desc for 56 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.092984]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.092989] pci 0000:80:02.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.113087] pci 0000:80:02.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.113106] pci 0000:80:02.2: PCI INT A -> GSI 56 (level, low) -> IRQ 56
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.141996] pci 0000:80:02.2: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142000] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142001] pci_bus 0000:00: resource 5 [io  0x1000-0xdfff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142003] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142004] pci_bus 0000:00: resource 7 [mem 0x90000000-0xebffefff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142006] pci_bus 0000:02: resource 0 [io  0x3000-0x3fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142007] pci_bus 0000:02: resource 1 [mem 0x91200000-0x912fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142009] pci_bus 0000:03: resource 0 [io  0x2000-0x2fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142010] pci_bus 0000:03: resource 1 [mem 0x91100000-0x911fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142012] pci_bus 0000:03: resource 2 [mem 0x91400000-0x914fffff pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142014] pci_bus 0000:05: resource 0 [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142015] pci_bus 0000:05: resource 1 [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142016] pci_bus 0000:06: resource 0 [io  0x1000-0x1fff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142018] pci_bus 0000:06: resource 1 [mem 0x90000000-0x910fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142019] pci_bus 0000:07: resource 4 [io  0x0000-0x0cf7]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142021] pci_bus 0000:07: resource 5 [io  0x1000-0xdfff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142022] pci_bus 0000:07: resource 6 [mem 0x000a0000-0x000bffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142023] pci_bus 0000:07: resource 7 [mem 0x90000000-0xebffefff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142026] pci_bus 0000:80: resource 4 [io  0x0000-0x0cf7]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142027] pci_bus 0000:80: resource 5 [io  0xe000-0xffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142028] pci_bus 0000:80: resource 6 [mem 0x000a0000-0x000bffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142029] pci_bus 0000:80: resource 7 [mem 0xec000000-0xfbffffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142031] pci_bus 0000:82: resource 1 [mem 0xec800000-0xec8fffff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142033] pci_bus 0000:82: resource 2 [mem 0xec000000-0xec7fffff 64bit pref]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.142303] NET: Registered protocol family 2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.153348] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.184013] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.204395] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.223139] TCP: Hash tables configured (established 524288 bind 65536)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.243485] TCP reno registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.262718] NET: Registered protocol family 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.272633] pci 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.303679] pci 0000:00:1a.0: PCI INT A disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.312616]   alloc irq_desc for 23 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.312618]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.312624] pci 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.343692] pci 0000:00:1d.0: PCI INT A disabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.352655] pci 0000:06:00.0: Boot video device
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.352792] Trying to unpack rootfs image as initramfs...
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.550338] Freeing initrd memory: 13549k freed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.557526] Simple Boot Flag at 0x44 set to 0x1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.584668] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.607688] VFS: Disk quotas dquot_6.5.2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.612782] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.633113] msgmni has been set to 32768
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.644034] alg: No test for stdrng (krng)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.662389] ksign: Installing public key data
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.673087] Loading keyring
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.682946] - Added public key E2105D8B18AD8BDF
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.702009] - User ID: Red Hat, Inc. (Kernel Module GPG key)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.713342] - Added public key D4A26C9CCD09BEDA
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.732604] - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.762685] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.783771] io scheduler noop registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.793419] io scheduler anticipatory registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.812604] io scheduler deadline registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.823433] io scheduler cfq registered (default)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843012] pcieport 0000:00:01.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843041]   alloc irq_desc for 77 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843042]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843049] pcieport 0000:00:01.0: irq 77 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843146] pcieport 0000:00:02.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843173]   alloc irq_desc for 78 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843174]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843181] pcieport 0000:00:02.0: irq 78 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843269] pcieport 0000:00:11.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843321]   alloc irq_desc for 79 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843322]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843331] pcieport 0000:00:11.0: irq 79 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843435] pcieport 0000:00:1c.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843466]   alloc irq_desc for 80 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843467]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843474] pcieport 0000:00:1c.0: irq 80 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843604] pcieport 0000:80:02.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843639]   alloc irq_desc for 81 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843642]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843652] pcieport 0000:80:02.0: irq 81 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843751] pcieport 0000:80:02.2: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843778]   alloc irq_desc for 82 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843779]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843786] pcieport 0000:80:02.2: irq 82 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843896] aer 0000:00:01.0:pcie02: service driver aer loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843934] aer 0000:00:02.0:pcie02: service driver aer loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.843971] aer 0000:00:11.0:pcie02: service driver aer loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.844003] aer 0000:80:02.0:pcie02: service driver aer loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.844032] aer 0000:80:02.2:pcie02: service driver aer loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.844039] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.862713] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.882864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.903667] intel_idle: MWAIT substates: 0x1120
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.903668] intel_idle: v0.4 model 0x3E
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.903669] intel_idle: lapic_timer_reliable_states 0xffffffff
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.903808] ipmi message handler version 39.2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.922087] IPMI System Interface driver.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.932713] ipmi_si: Adding ACPI-specified kcs state machine
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.952788] ipmi_si: Adding SMBIOS-specified kcs state machine: duplicate interface
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    5.974104] ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.003157] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x0, irq 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.081249] ipmi: Found new BMC (man_id: 0x0002a2,  prod_id: 0x008b, dev_id: 0x20)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.102947] IPMI kcs interface initialized
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.115261] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.142542] ACPI: Power Button [PWRF]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.152881] ACPI: acpi_idle yielding to intel_idle
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.214337] ERST: Table is not found!
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.222405] GHES: HEST is not enabled!
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.233909] Non-volatile memory driver v1.3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.243514] Linux agpgart interface v0.103
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.262342] crash memory driver: version 1.1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.273021] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.537282] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.553851] serial 00:0a: [io  0x03f8-0x03ff]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.553989] serial 00:0a: [irq 10]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.554384] serial 00:0a: activated
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.572205] Refined TSC clocksource calibration: 2600.022 MHz.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.572208] Switching to clocksource tsc
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.604648] brd: module loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.613840] loop: module loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.623069] input: Macintosh mouse button emulation as /devices/virtual/input/input1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.652786] Fixed MDIO Bus: probed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.662603] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.682987] ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.703560] ehci_hcd 0000:00:1a.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.703562] ehci_hcd 0000:00:1a.0: EHCI Host Controller
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.723239] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.744217] ehci_hcd 0000:00:1a.0: debug port 2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.767070] ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.767088] ehci_hcd 0000:00:1a.0: irq 16, io mem 0x91326000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.792245] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.803740] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.832618] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.853914] usb usb1: Product: EHCI Host Controller
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.872840] usb usb1: Manufacturer: Linux 2.6.32-358.23.2.el6.atlas.x86_64 ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.893916] usb usb1: SerialNumber: 0000:00:1a.0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.913090] usb usb1: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.932868] hub 1-0:1.0: USB hub found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.943221] hub 1-0:1.0: 2 ports detected
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.953826] ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.982929] ehci_hcd 0000:00:1d.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.982931] ehci_hcd 0000:00:1d.0: EHCI Host Controller
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    6.994297] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.023702] ehci_hcd 0000:00:1d.0: debug port 2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.046375] ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.046391] ehci_hcd 0000:00:1d.0: irq 23, io mem 0x91325000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.063351] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.074027] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.102706] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.123676] usb usb2: Product: EHCI Host Controller
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.142887] usb usb2: Manufacturer: Linux 2.6.32-358.23.2.el6.atlas.x86_64 ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.164178] usb usb2: SerialNumber: 0000:00:1d.0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.182989] usb usb2: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.202776] hub 2-0:1.0: USB hub found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.213002] hub 2-0:1.0: 2 ports detected
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.223757] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.243728] uhci_hcd: USB Universal Host Controller Interface driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.264130] PNP: No PS/2 controller found. Probing ports directly.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.284450] usb 1-1: new high speed USB device number 2 using ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.284942] mice: PS/2 mouse device common for all mice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.285064] rtc_cmos 00:08: RTC can wake from S4
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.285102] rtc_cmos 00:08: rtc core: registered rtc_cmos as rtc0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.285137] rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.285576] cpuidle: using governor ladder
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.403246] cpuidle: using governor menu
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.413043] EFI Variables Facility v0.08 2004-May-17
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.424514] usbcore: registered new interface driver hiddev
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.444176] usbcore: registered new interface driver usbhid
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.463853] usbhid: v2.6:USB HID core driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.483484] TCP cubic registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.493097] Initializing XFRM netlink socket
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.503646] NET: Registered protocol family 17
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.516861] usb 1-1: New USB device found, idVendor=8087, idProduct=0024
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.516863] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.516907] usb 1-1: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.517001] hub 1-1:1.0: USB hub found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.517108] hub 1-1:1.0: 6 ports detected
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.613040] registered taskstats version 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.619586] usb 2-1: new high speed USB device number 2 using ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.646905] rtc_cmos 00:08: setting system clock to 2014-03-31 18:09:02 UTC (1396289342)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.673781] Initalizing network drop monitor service
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.692814] Freeing unused kernel memory: 1264k freed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.704308] Write protecting the kernel read-only data: 10240k
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.723849] Freeing unused kernel memory: 908k freed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.733947] usb 2-1: New USB device found, idVendor=8087, idProduct=0024
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.733949] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.733992] usb 2-1: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.734086] hub 2-1:1.0: USB hub found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.734194] hub 2-1:1.0: 8 ports detected
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.808733] usb 1-1.6: new high speed USB device number 3 using ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.863831] Freeing unused kernel memory: 1676k freed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.884111] dracut: dracut-004-303.el6ccs
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.894635] usb 1-1.6: New USB device found, idVendor=0624, idProduct=0248
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.913796] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.943120] udev: starting version 147
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.953232] usb 1-1.6: Product: Gadget USB HUB
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.956141] pps_core: LinuxPPS API ver. 1 registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.956142] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.958365] PTP clock support registered
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.960616] dca service started, version 1.12.1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.967686] igb: Intel(R) Gigabit Ethernet Network Driver - version 4.0.1-k
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    7.967688] igb: Copyright (c) 2007-2012 Intel Corporation.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.083312] usb 1-1.6: Manufacturer: no manufacturer
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.102933] usb 1-1.6: SerialNumber: 0123456789
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.113721]   alloc irq_desc for 28 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.113722]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.113728] igb 0000:02:00.0: PCI INT B -> GSI 28 (level, low) -> IRQ 28
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.113799] usb 1-1.6: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.153874] igb 0000:02:00.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.153884] hub 1-1.6:1.0: USB hub found
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164487] hub 1-1.6:1.0: 5 ports detected
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164514]   alloc irq_desc for 83 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164515]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164519] igb 0000:02:00.0: irq 83 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164520]   alloc irq_desc for 84 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164521]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164524] igb 0000:02:00.0: irq 84 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164525]   alloc irq_desc for 85 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164526]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164529] igb 0000:02:00.0: irq 85 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164531]   alloc irq_desc for 86 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164532]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164535] igb 0000:02:00.0: irq 86 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164536]   alloc irq_desc for 87 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164537]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164541] igb 0000:02:00.0: irq 87 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164542]   alloc irq_desc for 88 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164542]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164545] igb 0000:02:00.0: irq 88 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164546]   alloc irq_desc for 89 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164547]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164550] igb 0000:02:00.0: irq 89 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164551]   alloc irq_desc for 90 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164552]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164555] igb 0000:02:00.0: irq 90 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164556]   alloc irq_desc for 91 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164557]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164559] igb 0000:02:00.0: irq 91 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.164599] igb 0000:02:00.0: PHY reset is blocked due to SOL/IDER session.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199737] igb 0000:02:00.0: added PHC on eth0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199738] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199740] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) f0:4d:a2:73:cc:ed
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199839] igb 0000:02:00.0: eth0: PBA No: 105000-000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199840] igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199891] igb 0000:02:00.3: PCI INT A -> GSI 26 (level, low) -> IRQ 26
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.199898] igb 0000:02:00.3: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205137]   alloc irq_desc for 92 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205139]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205143] igb 0000:02:00.3: irq 92 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205144]   alloc irq_desc for 93 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205145]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205148] igb 0000:02:00.3: irq 93 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205150]   alloc irq_desc for 94 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205151]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205154] igb 0000:02:00.3: irq 94 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205155]   alloc irq_desc for 95 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205156]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205159] igb 0000:02:00.3: irq 95 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205160]   alloc irq_desc for 96 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205160]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205164] igb 0000:02:00.3: irq 96 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205165]   alloc irq_desc for 97 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205165]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205168] igb 0000:02:00.3: irq 97 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205169]   alloc irq_desc for 98 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205170]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205174] igb 0000:02:00.3: irq 98 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205174]   alloc irq_desc for 99 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205175]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205181] igb 0000:02:00.3: irq 99 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205182]   alloc irq_desc for 100 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205183]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.205185] igb 0000:02:00.3: irq 100 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.262101] igb 0000:02:00.3: added PHC on eth1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.262102] igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.262104] igb 0000:02:00.3: eth1: (PCIe:5.0Gb/s:Width x4) f0:4d:a2:73:cc:ee
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.262176] igb 0000:02:00.3: eth1: PBA No: 105000-000
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.262178] igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.528762] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.545852] mlx4_core: Initializing 0000:82:00.0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.546137] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.574450]   alloc irq_desc for 58 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.574452]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.574459] mlx4_core 0000:82:00.0: PCI INT A -> GSI 58 (level, low) -> IRQ 58
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.603896] mlx4_core 0000:82:00.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.717232] usb 1-1.6.1: new high speed USB device number 4 using ehci_hcd
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.817994] usb 1-1.6.1: New USB device found, idVendor=0624, idProduct=0249
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.833726] usb 1-1.6.1: New USB device strings: Mfr=4, Product=5, SerialNumber=6
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.854816] usb 1-1.6.1: Product: Keyboard/Mouse Function
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.874180] usb 1-1.6.1: Manufacturer: Avocent
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.893364] usb 1-1.6.1: SerialNumber: 20121018
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.904198] usb 1-1.6.1: configuration #1 chosen from 1 choice
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.925170] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.0/input/input2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    8.964769] generic-usb 0003:0624:0249.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    9.014902] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.1/input/input3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    9.054160] generic-usb 0003:0624:0249.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    9.095185] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.2/input/input4
Mar 31 14:09:51 atlas-tds-mds1 kernel: [    9.134666] generic-usb 0003:0624:0249.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.212166] RPC: Registered named UNIX socket transport module.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.216584] RPC: Registered udp transport module.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.235737] RPC: Registered tcp transport module.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.255180] RPC: Registered tcp NFSv4.1 backchannel transport module.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.287797] Slow work thread pool: Starting up
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.296097] Slow work thread pool: Ready
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.306623] FS-Cache: Loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.342679] Registering the id_resolver key type
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.346157] FS-Cache: Netfs 'nfs' registered for caching
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.433028] NET: Registered protocol family 10
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.437098] lo: Disabled Privacy Extensions
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.457754] dracut: Mounted root filesystem 172.30.16.12:/images/nfsroot/hosts/atlas-tds-mds1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   14.498067] mlx4_core 0000:82:00.0: 64B EQEs/CQEs supported by the device but not enabled
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269791]   alloc irq_desc for 101 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269792]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269797] mlx4_core 0000:82:00.0: irq 101 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269799]   alloc irq_desc for 102 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269800]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269803] mlx4_core 0000:82:00.0: irq 102 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269804]   alloc irq_desc for 103 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269805]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269808] mlx4_core 0000:82:00.0: irq 103 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269811]   alloc irq_desc for 104 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269812]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269815] mlx4_core 0000:82:00.0: irq 104 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269816]   alloc irq_desc for 105 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269817]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269820] mlx4_core 0000:82:00.0: irq 105 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269821]   alloc irq_desc for 106 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269822]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269827] mlx4_core 0000:82:00.0: irq 106 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269828]   alloc irq_desc for 107 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269829]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269832] mlx4_core 0000:82:00.0: irq 107 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269833]   alloc irq_desc for 108 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269834]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269837] mlx4_core 0000:82:00.0: irq 108 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269838]   alloc irq_desc for 109 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269839]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269842] mlx4_core 0000:82:00.0: irq 109 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269844]   alloc irq_desc for 110 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269845]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269847] mlx4_core 0000:82:00.0: irq 110 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269849]   alloc irq_desc for 111 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269850]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269853] mlx4_core 0000:82:00.0: irq 111 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269854]   alloc irq_desc for 112 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269855]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269858] mlx4_core 0000:82:00.0: irq 112 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269859]   alloc irq_desc for 113 on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269860]   alloc kstat_irqs on node 1
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.269863] mlx4_core 0000:82:00.0: irq 113 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.324062] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   15.534425] dracut: Switching root
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   25.149648] eth0: no IPv6 routers present
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.343065] udev: starting version 147
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.449041] iTCO_vendor_support: vendor-support=0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.462021] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.478506] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.517188] ahci 0000:00:1f.2: version 3.0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.517254]   alloc irq_desc for 19 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.517256]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.517262] ahci 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.538242]   alloc irq_desc for 114 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.538243]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.538267] ahci 0000:00:1f.2: irq 114 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.548935] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.569618] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.599098] ahci 0000:00:1f.2: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.609022] scsi0 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.610978] scsi1 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.618974] scsi2 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.629010] scsi3 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.638661] scsi4 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.648280] scsi5 : ahci
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.650019] ata1: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324100 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.679119] ata2: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324180 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.708392] ata3: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324200 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.728949] ata4: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324280 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.749713] ata5: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324300 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   48.778918] ata6: SATA max UDMA/133 abar m2048@0x91324000 port 0x91324380 irq 114
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104141] ata6: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104177] ata1: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104181] ata3: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104199] ata2: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104208] ata5: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.104216] ata4: SATA link down (SStatus 0 SControl 300)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.199515] mlx4_core 0000:82:00.0: vpd r/w failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update.
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.215112] i801_smbus 0000:00:1f.3: PCI INT C -> GSI 19 (level, low) -> IRQ 19
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.229160] ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.259565] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.479878] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.506487] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.519266] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.545886] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.559607] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.593431] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.608375] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.633527] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.648949] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.674226] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.689291] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.714664] microcode: CPU5 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.729832] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.763218] microcode: CPU6 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.769962] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.803433] microcode: CPU7 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.818634] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.843956] microcode: CPU8 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.859249] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.884723] microcode: CPU9 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.899792] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.933427] microcode: CPU10 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.948480] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.973591] microcode: CPU11 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   49.989072] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.013964] microcode: CPU12 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.029487] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.054732] microcode: CPU13 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.070185] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.103524] microcode: CPU14 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.118950] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.143965] udev: renamed network interface eth1 to em2
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.144959] microcode: CPU15 sig=0x306e4, pf=0x1, revision=0x416
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.144963] platform microcode: firmware: requesting intel-ucode/06-3e-04
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.149881] Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.157727] megasas: 06.504.01.00-rh1 Mon. Oct. 8 17:00:00 PDT 2012
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.164567] megasas: 0x1000:0x005b:0x1000:0x9265: bus 3:slot 0:func 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.164580] megaraid_sas 0000:03:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.164583] megaraid_sas 0000:03:00.0: setting latency timer to 64
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.290083] megasas: FW now in Ready state
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308820]   alloc irq_desc for 115 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308822]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308828] megaraid_sas 0000:03:00.0: irq 115 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308829]   alloc irq_desc for 116 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308830]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308833] megaraid_sas 0000:03:00.0: irq 116 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308834]   alloc irq_desc for 117 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308835]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308848] megaraid_sas 0000:03:00.0: irq 117 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308849]   alloc irq_desc for 118 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308850]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308853] megaraid_sas 0000:03:00.0: irq 118 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308854]   alloc irq_desc for 119 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308855]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308858] megaraid_sas 0000:03:00.0: irq 119 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308859]   alloc irq_desc for 120 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308860]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308863] megaraid_sas 0000:03:00.0: irq 120 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308865]   alloc irq_desc for 121 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308865]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308868] megaraid_sas 0000:03:00.0: irq 121 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308870]   alloc irq_desc for 122 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308870]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308874] megaraid_sas 0000:03:00.0: irq 122 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308875]   alloc irq_desc for 123 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308876]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308879] megaraid_sas 0000:03:00.0: irq 123 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308880]   alloc irq_desc for 124 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308881]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308884] megaraid_sas 0000:03:00.0: irq 124 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308886]   alloc irq_desc for 125 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308886]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308890] megaraid_sas 0000:03:00.0: irq 125 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308891]   alloc irq_desc for 126 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308892]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308895] megaraid_sas 0000:03:00.0: irq 126 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308896]   alloc irq_desc for 127 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308897]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308900] megaraid_sas 0000:03:00.0: irq 127 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308901]   alloc irq_desc for 128 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308902]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308905] megaraid_sas 0000:03:00.0: irq 128 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308906]   alloc irq_desc for 129 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308907]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308910] megaraid_sas 0000:03:00.0: irq 129 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308911]   alloc irq_desc for 130 on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308912]   alloc kstat_irqs on node 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.308915] megaraid_sas 0000:03:00.0: irq 130 for MSI/MSI-X
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.330619] megasas:IOC Init cmd success
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.359641] megasas: INIT adapter done
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.431743] scsi6 : LSI SAS based MegaRAID driver
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.440037] scsi 6:0:0:0: Direct-Access     LSI      MR9265-8i        3.16 PQ: 0 ANSI: 5
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.482357] scsi 6:2:0:0: Direct-Access     LSI      MR9265-8i        3.16 PQ: 0 ANSI: 5
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.516868] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.544846] ACPI: WMI: Mapper loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.637097] sd 6:2:0:0: [sda] 1952448512 512-byte logical blocks: (999 GB/931 GiB)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.650594] sd 6:2:0:0: [sda] Write Protect is off
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.669706] sd 6:2:0:0: [sda] Mode Sense: 1f 00 00 08
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.669770] sd 6:2:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.700271]  sda: sda1 sda2 sda3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.711022] sd 6:2:0:0: [sda] Attached SCSI disk
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   50.806964] sd 6:2:0:0: Attached scsi generic sg0 type 0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.121077] device-mapper: uevent: version 1.0.3
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.129847] device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.166702] device-mapper: multipath: version 1.5.0 loaded
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.599584] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.914018] ip_tables: (C) 2000-2006 Netfilter Core Team
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   51.976774] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   52.159206] 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   52.179489] All bugs added by David S. Miller <davem@redhat.com>
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   52.199510] 8021q: adding VLAN 0 to HW filter on device eth0
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   52.353746] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   52.354301] ADDRCONF(NETDEV_UP): ib0: link is not ready
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   54.391751] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:09:51 atlas-tds-mds1 kernel: [   57.068196] ipmi device interface
Mar 31 14:09:53 atlas-tds-mds1 kernel: [   58.413301] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:10:01 atlas-tds-mds1 kernel: [   66.436656] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:10:10 atlas-tds-mds1 kernel: [   75.456122] netlink: 12 bytes leftover after parsing attributes.
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.468005] Fusion MPT base driver 3.04.20
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.470580] Copyright (c) 1999-2008 LSI Corporation
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.494077] Fusion MPT misc device (ioctl) driver 3.04.20
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.501193] mptctl: Registered with Fusion MPT base driver
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.520834] mptctl: /dev/mptctl @ (major,minor=10,220)
Mar 31 14:10:15 atlas-tds-mds1 kernel: [   80.576190] mpt2sas version 13.101.00.00 loaded
Mar 31 14:10:17 atlas-tds-mds1 kernel: [   82.462880] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:10:32 atlas-tds-mds1 kernel: [   97.709646] netlink: 12 bytes leftover after parsing attributes.
Mar 31 14:10:33 atlas-tds-mds1 kernel: [   98.485247] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
Mar 31 14:10:37 atlas-tds-mds1 kernel: [  102.195095] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Mar 31 14:10:47 atlas-tds-mds1 kernel: [  112.490408] ib0: no IPv6 routers present
Mar 31 14:11:05 atlas-tds-mds1 kernel: [  130.335705] LNet: HW CPU cores: 16, npartitions: 4
Mar 31 14:11:05 atlas-tds-mds1 kernel: [  130.341333] alg: No test for crc32 (crc32-table)
Mar 31 14:11:05 atlas-tds-mds1 kernel: [  130.358344] alg: No test for adler32 (adler32-zlib)
Mar 31 14:11:05 atlas-tds-mds1 kernel: [  130.369373] alg: No test for crc32 (crc32-pclmul)
Mar 31 14:11:09 atlas-tds-mds1 kernel: [  134.408640] padlock: VIA PadLock Hash Engine not detected.
Mar 31 14:11:13 atlas-tds-mds1 kernel: [  138.678710] LNet: Added LNI 10.36.226.79@o2ib [63/2560/0/180]
Mar 31 14:11:14 atlas-tds-mds1 kernel: [  139.207430] LNet: Added LNI 10.36.226.79@o2ib200 [63/2560/0/180]
Mar 31 14:11:14 atlas-tds-mds1 kernel: [  139.403610] Lustre: Lustre: Build Version: 2.4.3-gd00e4d6-CHANGED-2.6.32-358.23.2.el6.atlas.x86_64
Mar 31 14:11:15 atlas-tds-mds1 kernel: [  140.842279] LDISKFS-fs (sda1): recovery complete
Mar 31 14:11:15 atlas-tds-mds1 kernel: [  140.853414] LDISKFS-fs (sda1): mounted filesystem with ordered data mode. quota=on. Opts: 
Mar 31 14:12:02 atlas-tds-mds1 kernel: [  187.985940] LDISKFS-fs (sda3): recovery complete
Mar 31 14:12:02 atlas-tds-mds1 kernel: [  188.026442] LDISKFS-fs (sda3): mounted filesystem with ordered data mode. quota=on. Opts: 
Mar 31 14:12:05 atlas-tds-mds1 kernel: [  190.263659] Lustre: atlastds-MDT0000: used disk, loading
Mar 31 14:12:05 atlas-tds-mds1 kernel: [  190.645660] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.461317] LustreError: 11-0: atlastds-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.486630] LustreError: 11922:0:(mdt_handler.c:6024:mdt_iocontrol()) atlastds-MDT0000: Aborting recovery for device
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.486634] LustreError: 11922:0:(ldlm_lib.c:2137:target_stop_recovery_thread()) atlastds-MDT0000: Aborting recovery
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.486642] Lustre: 15945:0:(ldlm_lib.c:1801:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.486853] Lustre: atlastds-MDT0000: disconnecting 83 stale clients
Mar 31 14:12:06 atlas-tds-mds1 kernel: [  191.487941] Lustre: 15945:0:(ldlm_lib.c:1801:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Mar 31 14:14:36 atlas-tds-mds1 kernel: [  341.521314] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:17:06 atlas-tds-mds1 kernel: [  491.566647] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.057997] LustreError: 12309:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed: 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058059] LustreError: 12460:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed: 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058062] LustreError: 12460:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) LBUG
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058064] Pid: 12460, comm: mdt01_138
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058064] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058065] Call Trace:
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058083]  [<ffffffffa03dd895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058091]  [<ffffffffa03dde97>] lbug_with_loc+0x47/0xb0 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058104]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058113] LustreError: 13083:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed: 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058116]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058118] LustreError: 13083:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) LBUG
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058120] Pid: 13083, comm: mdt03_249
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058121] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058121] Call Trace:
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058126]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058142]  [<ffffffffa03dd895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058150]  [<ffffffffa03dde97>] lbug_with_loc+0x47/0xb0 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058160]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058166]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058176]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058185]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058187]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058196]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058205]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058221]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058232]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058246]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058248]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058257]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058266]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058273]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058282]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058293]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058294]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058298] LustreError: 13062:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed: 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058301] LustreError: 13062:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) LBUG
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058302] Pid: 13062, comm: mdt03_228
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058303] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058304] Call Trace:
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058309]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058314]  [<ffffffffa03dd895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058319]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058322]  [<ffffffffa03dde97>] lbug_with_loc+0x47/0xb0 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058323]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058332]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058336]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058341]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058345]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058348]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058350]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058356]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058371]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058373]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058376]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058381]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058386]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058395]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058399]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058405]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058411]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058414]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058423]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058425]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058425] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058435]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058438]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058439]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058450]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058462]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058476]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058483] LustreError: dumping log to /tmp/lustre-log.1396289899.12460
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058485]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058487]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058488]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058489] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058496]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058522]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058525]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058532] LustreError: dumping log to /tmp/lustre-log.1396289899.13083
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058550]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058573]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058576]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058599]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058622]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058623]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058624] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  565.058642] LustreError: dumping log to /tmp/lustre-log.1396289899.13062
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.852536] LustreError: 12309:0:(mdt_handler.c:3615:mdt_intent_lock_replace()) LBUG
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.882193] Pid: 12309, comm: mdt00_243
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.882238] LustreError: can't open /tmp/lustre-log.1396289899.13062 for dump: rc -17
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.922065] 
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.922065] Call Trace:
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.932235]  [<ffffffffa03dd895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.961134]  [<ffffffffa03dde97>] lbug_with_loc+0x47/0xb0 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  566.981093]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.001686]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.021845]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.042315]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.062719]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.091534]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.111347]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.131754]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.152639]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.172441]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.202137]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.221957]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.242608]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.271351]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.282485]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:18:21 atlas-tds-mds1 kernel: [  567.302809]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.331223]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.342374]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.362504]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.382521]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.402008] 
Mar 31 14:18:22 atlas-tds-mds1 kernel: [  567.411288] LustreError: dumping log to /tmp/lustre-log.1396289902.12309
Mar 31 14:19:36 atlas-tds-mds1 kernel: [  641.619767] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.814791] INFO: task mdt00_243:12309 blocked for more than 120 seconds.
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.830560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.851897] mdt00_243     D 0000000000000001     0 12309      2 0x00000000
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.880300]  ffff881002a0fb60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.901812]  0000000000000000 ffffffffa0e0f580 ffff881002a0fb20 ffff880767b38000
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.930297]  ffff881002a045f8 ffff881002a0ffd8 000000000000fb88 ffff881002a045f8
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.951573] Call Trace:
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.960803]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  725.981017]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.001081]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.021546]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.041703]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.070309]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.090745]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.111250]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.131084]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.151476]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.180751]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.200635]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.221840]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.241632]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.262032]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.290783]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.301896]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.330435]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:00 atlas-tds-mds1 kernel: [  726.350663]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.361736]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.381889]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.410471]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.421548] INFO: task mdt01_138:12460 blocked for more than 120 seconds.
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.450442] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.471743] mdt01_138     D 0000000000000007     0 12460      2 0x00000000
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.491972]  ffff881002453b60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.521589]  0000000000000000 ffffffffa0e0f580 ffff881002453b20 ffff880767b57000
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.542549]  ffff881002451af8 ffff881002453fd8 000000000000fb88 ffff881002451af8
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.571410] Call Trace:
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.580633]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.600750]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.620877]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.641455]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.661617]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.681803]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.702137]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.730941]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.751000]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.771315]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.792277]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.820643]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.841870]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.861610]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.882182]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.910809]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.922098]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.950736]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.970857]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  726.981820]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.001975]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.022262]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.041702] INFO: task mdt03_228:13062 blocked for more than 120 seconds.
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.062081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.091841] mdt03_228     D 000000000000000f     0 13062      2 0x00000000
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.112100]  ffff880ffd023b60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.141476]  0000000000000000 ffffffffa0e0f580 ffff880ffd023b20 ffff880841000400
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.162485]  ffff880ffd021ab8 ffff880ffd023fd8 000000000000fb88 ffff880ffd021ab8
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.191509] Call Trace:
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.200732]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.220746]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.240908]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.261324]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.281487]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.301776]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.322370]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.351404]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:21:01 atlas-tds-mds1 kernel: [  727.371225]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.391554]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.412673]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.432265]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.461788]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.481621]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.502010]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.530838]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.542118]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.562337]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.582311]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.601804]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.621993]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.642328]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.661837] INFO: task mdt03_249:13083 blocked for more than 120 seconds.
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.681997] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.711846] mdt03_249     D 000000000000000e     0 13083      2 0x00000000
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.732018]  ffff880ffd3afb60 0000000000000046 ffff880ffd3afb28 ffff880ffd3afb24
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.761703]  0000000000000000 ffff88087fe72b00 ffff88089c416740 0000000000000400
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.782547]  ffff880ffd3adaf8 ffff880ffd3affd8 000000000000fb88 ffff880ffd3adaf8
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.811692] Call Trace:
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.813521]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.840999]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.861039]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.881621]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.901897]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.922142]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.942598]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.971633]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  727.991454]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.011665]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.032414]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.052323]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.081834]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.101619]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.122169]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.142492]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.162233]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.182310]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.202457]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.221909]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.242044]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:21:02 atlas-tds-mds1 kernel: [  728.262238]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:22:06 atlas-tds-mds1 kernel: [  791.677976] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 14:23:02 atlas-tds-mds1 kernel: [  848.327822] INFO: task mdt00_243:12309 blocked for more than 120 seconds.
Mar 31 14:23:02 atlas-tds-mds1 kernel: [  848.346191] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:23:02 atlas-tds-mds1 kernel: [  848.367635] mdt00_243     D 0000000000000001     0 12309      2 0x00000000
Mar 31 14:23:02 atlas-tds-mds1 kernel: [  848.396031]  ffff881002a0fb60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.417193]  0000000000000000 ffffffffa0e0f580 ffff881002a0fb20 ffff880767b38000
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.445874]  ffff881002a045f8 ffff881002a0ffd8 000000000000fb88 ffff881002a045f8
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.467189] Call Trace:
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.476437]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.497012]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.516738]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.537183]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.557210]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.585965]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.606417]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.626717]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.646616]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.667189]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.696450]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.716285]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.737571]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.757197]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.786269]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.806619]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.826208]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.846245]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.866446]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.877467]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.906264]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.926410]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.937354] INFO: task mdt01_138:12460 blocked for more than 120 seconds.
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.966486] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  848.998510] mdt01_138     D 0000000000000007     0 12460      2 0x00000000
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.016519]  ffff881002453b60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.037746]  0000000000000000 ffffffffa0e0f580 ffff881002453b20 ffff880767b57000
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.066692]  ffff881002451af8 ffff881002453fd8 000000000000fb88 ffff881002451af8
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.087573] Call Trace:
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.096690]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.116715]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.136954]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.157404]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.177561]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.206379]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.226754]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.247565]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.267203]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.287821]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.316971]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.336972]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.358178]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.386255]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.406836]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:23:03 atlas-tds-mds1 kernel: [  849.427178]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.446661]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.466865]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.487035]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.506507]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.526744]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.546864]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.566383] INFO: task mdt03_228:13062 blocked for more than 120 seconds.
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.586677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.608263] mdt03_228     D 000000000000000f     0 13062      2 0x00000000
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.636677]  ffff880ffd023b60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.658152]  0000000000000000 ffffffffa0e0f580 ffff880ffd023b20 ffff880841000400
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.687113]  ffff880ffd021ab8 ffff880ffd023fd8 000000000000fb88 ffff880ffd021ab8
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.707989] Call Trace:
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.717137]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.737093]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.757319]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.777954]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.806481]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.826630]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.847010]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.867680]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.887671]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.907853]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.937010]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.956876]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  849.978279]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.006492]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.027114]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.047517]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.066958]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.087095]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.107251]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.126685]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.146755]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.166939]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.186501] INFO: task mdt03_249:13083 blocked for more than 120 seconds.
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.206937] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.228177] mdt03_249     D 000000000000000e     0 13083      2 0x00000000
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.256587]  ffff880ffd3afb60 0000000000000046 ffff880ffd3afb28 ffff880ffd3afb24
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.278039]  0000000000000000 ffff88087fe72b00 ffff88089c416740 0000000000000400
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.307172]  ffff880ffd3adaf8 ffff880ffd3affd8 000000000000fb88 ffff880ffd3adaf8
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.327945] Call Trace:
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.337147]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.357377]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.377520]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.397932]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:23:04 atlas-tds-mds1 kernel: [  850.418049]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.446827]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.467223]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.487741]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.507741]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.528079]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.557496]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.577282]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.607099]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.626732]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.647276]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.667775]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.687320]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.707486]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.727485]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.746916]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.767148]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:23:05 atlas-tds-mds1 kernel: [  850.787204]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:24:36 atlas-tds-mds1 kernel: [  941.734812] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.844601] INFO: task mdt00_243:12309 blocked for more than 120 seconds.
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.861876] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.883364] mdt00_243     D 0000000000000001     0 12309      2 0x00000000
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.912127]  ffff881002a0fb60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.933454]  0000000000000000 ffffffffa0e0f580 ffff881002a0fb20 ffff880767b38000
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.962475]  ffff881002a045f8 ffff881002a0ffd8 000000000000fb88 ffff881002a045f8
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.983055] Call Trace:
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  970.992486]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.012476]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.032523]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.053130]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.081750]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.102013]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.122338]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.143011]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.162849]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.191662]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.212570]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.232632]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.262187]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.282112]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.302633]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.322933]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.342646]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.362623]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.382672]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.402131]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.422492]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.442550]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:25:05 atlas-tds-mds1 kernel: [  971.461885] INFO: task mdt01_138:12460 blocked for more than 120 seconds.
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.482470] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.512091] mdt01_138     D 0000000000000007     0 12460      2 0x00000000
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.532301]  ffff881002453b60 0000000000000046 0000000000000000 0000000000000001
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.553704]  0000000000000000 ffffffffa0e0f580 ffff881002453b20 ffff880767b57000
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.582697]  ffff881002451af8 ffff881002453fd8 000000000000fb88 ffff881002451af8
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.603547] Call Trace:
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.612683]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.632907]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.653052]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.681799]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.701818]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.722206]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.742680]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.763190]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.782997]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.803288]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.832630]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.852646]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.882250]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.902070]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.922317]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.943044]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.962522]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  971.982715]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  972.002906]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  972.022128]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  972.042313]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:25:06 atlas-tds-mds1 kernel: [  972.062530]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:27:06 atlas-tds-mds1 kernel: [ 1091.794143] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 14:29:36 atlas-tds-mds1 kernel: [ 1241.825816] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:32:06 atlas-tds-mds1 kernel: [ 1391.886213] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 14:32:50 atlas-tds-mds1 kernel: [ 1436.115858] Lustre: 12321:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply
Mar 31 14:32:50 atlas-tds-mds1 kernel: [ 1436.115860]   req@ffff880767b38000 x1463845698693680/t64424509473(0) o101->6a5c48ea-8b4f-d046-431e-52c0fb4076be@2@gni2:0/0 lens 656/2144 e 3 to 0 dl 1396290775 ref 2 fl Interpret:/2/0 rc 0/0
Mar 31 14:32:50 atlas-tds-mds1 kernel: [ 1436.118854] Lustre: 12443:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply
Mar 31 14:32:50 atlas-tds-mds1 kernel: [ 1436.118856]   req@ffff880767b57000 x1463845698668116/t64424509472(0) o101->411700ef-c5bd-ec9f-cb66-dc45aad45f27@28@gni2:0/0 lens 656/2144 e 3 to 0 dl 1396290775 ref 2 fl Interpret:/2/0 rc 0/0
Mar 31 14:32:50 atlas-tds-mds1 kernel: [ 1436.306053] Lustre: 12321:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages
Mar 31 14:34:36 atlas-tds-mds1 kernel: [ 1541.941687] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:36:03 atlas-tds-mds1 kernel: [ 1629.488568] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 14:36:03 atlas-tds-mds1 kernel: [ 1629.491573] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 14:36:03 atlas-tds-mds1 kernel: [ 1629.491578] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:36:03 atlas-tds-mds1 kernel: [ 1629.588375] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:36:03 atlas-tds-mds1 kernel: [ 1629.637243] Lustre: Skipped 2 previous similar messages
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.466398] LNet: Service thread pid 13062 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.518388] Pid: 13062, comm: mdt03_228
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.528237] 
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.528238] Call Trace:
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.539245]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.559084]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.587494]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.607893]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.628391]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.648784]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.669111]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.697935]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:38:19 atlas-tds-mds1 kernel: [ 1765.717748]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.738210]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.759146]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.779142]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.808583]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.828377]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.848713]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.877609]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.888734]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.908746]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.937626]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.948577]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.968754]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1765.988933]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.008569] 
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.017784] LustreError: dumping log to /tmp/lustre-log.1396291100.13062
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.039769] LNet: Service thread pid 13083 was inactive for 1200.57s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.098463] Pid: 13083, comm: mdt03_249
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.108136] 
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.108137] Call Trace:
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.118604]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.138488]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.158821]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.179150]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.207911]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.228101]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.248518]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.269008]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.289040]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.317772]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.338894]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.358781]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.388347]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.408183]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.428773]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.449093]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.468597]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.488855]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.508838]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.528308]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.548287]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.568319]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.587878] 
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.589171] LNet: Service thread pid 12460 was inactive for 1201.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.649109] Pid: 12460, comm: mdt01_138
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.658638] 
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.658639] Call Trace:
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.669599]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.689209]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:38:20 atlas-tds-mds1 kernel: [ 1766.709194]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.738308]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.758490]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.778830]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.799358]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.828217]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.847986]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.868186]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.889108]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.908924]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.938560]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.958405]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.978773]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1766.999269]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.018940]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.039066]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.059149]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.078515]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.098666]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.118814]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.138252] 
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.139530] Pid: 12309, comm: mdt00_243
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.158081] 
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.158082] Call Trace:
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.168629]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.188666]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.208852]  [<ffffffffa0db05a6>] mdt_intent_lock_replace+0x286/0x400 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.229515]  [<ffffffffa0db6387>] mdt_intent_reint+0x477/0x520 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.249680]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.278154]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.298546]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.319082]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.338919]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.359316]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.388712]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.408547]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.429808]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.449477]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.478385]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.498830]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.518408]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.538674]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.558880]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.578291]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.598474]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.618409]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 14:38:21 atlas-tds-mds1 kernel: [ 1767.629670] 
Mar 31 14:38:33 atlas-tds-mds1 kernel: [ 1779.558766] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 14:38:33 atlas-tds-mds1 kernel: [ 1779.565454] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:38:33 atlas-tds-mds1 kernel: [ 1779.624040] Lustre: Skipped 3 previous similar messages
Mar 31 14:39:36 atlas-tds-mds1 kernel: [ 1842.012472] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:39:36 atlas-tds-mds1 kernel: [ 1842.046567] LustreError: Skipped 1 previous similar message
Mar 31 14:41:03 atlas-tds-mds1 kernel: [ 1929.626289] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 14:41:03 atlas-tds-mds1 kernel: [ 1929.637384] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:41:03 atlas-tds-mds1 kernel: [ 1929.637387] Lustre: Skipped 3 previous similar messages
Mar 31 14:41:03 atlas-tds-mds1 kernel: [ 1929.708812] Lustre: Skipped 3 previous similar messages
Mar 31 14:43:33 atlas-tds-mds1 kernel: [ 2079.699658] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 14:43:33 atlas-tds-mds1 kernel: [ 2079.700525] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:43:33 atlas-tds-mds1 kernel: [ 2079.700528] Lustre: Skipped 3 previous similar messages
Mar 31 14:43:33 atlas-tds-mds1 kernel: [ 2079.784768] Lustre: Skipped 3 previous similar messages
Mar 31 14:46:03 atlas-tds-mds1 kernel: [ 2229.733154] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 14:46:03 atlas-tds-mds1 kernel: [ 2229.735616] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:46:03 atlas-tds-mds1 kernel: [ 2229.735619] Lustre: Skipped 3 previous similar messages
Mar 31 14:46:03 atlas-tds-mds1 kernel: [ 2229.820802] Lustre: Skipped 3 previous similar messages
Mar 31 14:48:33 atlas-tds-mds1 kernel: [ 2379.807765] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 14:48:33 atlas-tds-mds1 kernel: [ 2379.818453] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:48:33 atlas-tds-mds1 kernel: [ 2379.818455] Lustre: Skipped 3 previous similar messages
Mar 31 14:48:33 atlas-tds-mds1 kernel: [ 2379.896919] Lustre: Skipped 3 previous similar messages
Mar 31 14:49:36 atlas-tds-mds1 kernel: [ 2442.244141] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 14:49:36 atlas-tds-mds1 kernel: [ 2442.279913] LustreError: Skipped 3 previous similar messages
Mar 31 14:51:03 atlas-tds-mds1 kernel: [ 2529.846445] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 14:51:03 atlas-tds-mds1 kernel: [ 2529.849106] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:51:03 atlas-tds-mds1 kernel: [ 2529.849109] Lustre: Skipped 3 previous similar messages
Mar 31 14:51:03 atlas-tds-mds1 kernel: [ 2529.932767] Lustre: Skipped 3 previous similar messages
Mar 31 14:53:33 atlas-tds-mds1 kernel: [ 2679.956496] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 14:53:33 atlas-tds-mds1 kernel: [ 2679.956718] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:53:33 atlas-tds-mds1 kernel: [ 2679.956720] Lustre: Skipped 3 previous similar messages
Mar 31 14:53:33 atlas-tds-mds1 kernel: [ 2680.039281] Lustre: Skipped 3 previous similar messages
Mar 31 14:56:03 atlas-tds-mds1 kernel: [ 2829.955913] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 14:56:03 atlas-tds-mds1 kernel: [ 2829.957845] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:56:03 atlas-tds-mds1 kernel: [ 2829.957848] Lustre: Skipped 3 previous similar messages
Mar 31 14:56:03 atlas-tds-mds1 kernel: [ 2830.044444] Lustre: Skipped 3 previous similar messages
Mar 31 14:58:33 atlas-tds-mds1 kernel: [ 2980.002900] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 14:58:33 atlas-tds-mds1 kernel: [ 2980.010339] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 14:58:33 atlas-tds-mds1 kernel: [ 2980.010342] Lustre: Skipped 3 previous similar messages
Mar 31 14:58:33 atlas-tds-mds1 kernel: [ 2980.090451] Lustre: Skipped 3 previous similar messages
Mar 31 15:02:06 atlas-tds-mds1 kernel: [ 3192.506439] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 15:02:06 atlas-tds-mds1 kernel: [ 3192.539666] LustreError: Skipped 4 previous similar messages
Mar 31 15:03:33 atlas-tds-mds1 kernel: [ 3280.105144] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 15:03:33 atlas-tds-mds1 kernel: [ 3280.116852] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:03:33 atlas-tds-mds1 kernel: [ 3280.116855] Lustre: Skipped 7 previous similar messages
Mar 31 15:03:33 atlas-tds-mds1 kernel: [ 3280.192092] Lustre: Skipped 7 previous similar messages
Mar 31 15:13:33 atlas-tds-mds1 kernel: [ 3880.333109] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 15:13:33 atlas-tds-mds1 kernel: [ 3880.344776] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:13:33 atlas-tds-mds1 kernel: [ 3880.344779] Lustre: Skipped 15 previous similar messages
Mar 31 15:13:33 atlas-tds-mds1 kernel: [ 3880.416031] Lustre: Skipped 15 previous similar messages
Mar 31 15:14:36 atlas-tds-mds1 kernel: [ 3942.784000] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 15:14:36 atlas-tds-mds1 kernel: [ 3942.819166] LustreError: Skipped 4 previous similar messages
Mar 31 15:24:36 atlas-tds-mds1 kernel: [ 4543.020710] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 15:24:36 atlas-tds-mds1 kernel: [ 4543.053304] LustreError: Skipped 3 previous similar messages
Mar 31 15:26:03 atlas-tds-mds1 kernel: [ 4630.615263] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 15:26:03 atlas-tds-mds1 kernel: [ 4630.616519] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:26:03 atlas-tds-mds1 kernel: [ 4630.616522] Lustre: Skipped 19 previous similar messages
Mar 31 15:26:03 atlas-tds-mds1 kernel: [ 4630.705405] Lustre: Skipped 19 previous similar messages
Mar 31 15:36:03 atlas-tds-mds1 kernel: [ 5230.834046] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 15:36:03 atlas-tds-mds1 kernel: [ 5230.859634] Lustre: Skipped 15 previous similar messages
Mar 31 15:36:03 atlas-tds-mds1 kernel: [ 5230.870489] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:36:03 atlas-tds-mds1 kernel: [ 5230.919885] Lustre: Skipped 18 previous similar messages
Mar 31 15:37:06 atlas-tds-mds1 kernel: [ 5293.299292] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 15:37:06 atlas-tds-mds1 kernel: [ 5293.332607] LustreError: Skipped 4 previous similar messages
Mar 31 15:46:03 atlas-tds-mds1 kernel: [ 5831.103513] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 15:46:03 atlas-tds-mds1 kernel: [ 5831.112192] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:46:03 atlas-tds-mds1 kernel: [ 5831.112195] Lustre: Skipped 14 previous similar messages
Mar 31 15:46:03 atlas-tds-mds1 kernel: [ 5831.193439] Lustre: Skipped 15 previous similar messages
Mar 31 15:49:36 atlas-tds-mds1 kernel: [ 6043.551186] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 15:49:36 atlas-tds-mds1 kernel: [ 6043.592464] LustreError: Skipped 4 previous similar messages
Mar 31 15:58:33 atlas-tds-mds1 kernel: [ 6581.323619] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 15:58:33 atlas-tds-mds1 kernel: [ 6581.325263] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 15:58:33 atlas-tds-mds1 kernel: [ 6581.325266] Lustre: Skipped 17 previous similar messages
Mar 31 15:58:33 atlas-tds-mds1 kernel: [ 6581.412722] Lustre: Skipped 19 previous similar messages
Mar 31 16:02:06 atlas-tds-mds1 kernel: [ 6793.824643] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 16:02:06 atlas-tds-mds1 kernel: [ 6793.862156] LustreError: Skipped 4 previous similar messages
Mar 31 16:08:33 atlas-tds-mds1 kernel: [ 7181.544025] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 16:08:33 atlas-tds-mds1 kernel: [ 7181.563026] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 16:08:33 atlas-tds-mds1 kernel: [ 7181.563029] Lustre: Skipped 15 previous similar messages
Mar 31 16:08:33 atlas-tds-mds1 kernel: [ 7181.626876] Lustre: Skipped 15 previous similar messages
Mar 31 16:14:36 atlas-tds-mds1 kernel: [ 7544.096905] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 16:14:36 atlas-tds-mds1 kernel: [ 7544.131675] LustreError: Skipped 4 previous similar messages
Mar 31 16:18:33 atlas-tds-mds1 kernel: [ 7781.828201] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 16:18:33 atlas-tds-mds1 kernel: [ 7781.845616] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 16:18:33 atlas-tds-mds1 kernel: [ 7781.845618] Lustre: Skipped 16 previous similar messages
Mar 31 16:18:33 atlas-tds-mds1 kernel: [ 7781.910651] Lustre: Skipped 14 previous similar messages
Mar 31 16:24:36 atlas-tds-mds1 kernel: [ 8144.332809] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 16:24:36 atlas-tds-mds1 kernel: [ 8144.365745] LustreError: Skipped 3 previous similar messages
Mar 31 16:28:33 atlas-tds-mds1 kernel: [ 8382.048174] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 16:28:33 atlas-tds-mds1 kernel: [ 8382.074170] Lustre: Skipped 16 previous similar messages
Mar 31 16:28:33 atlas-tds-mds1 kernel: [ 8382.093590] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 16:28:33 atlas-tds-mds1 kernel: [ 8382.134816] Lustre: Skipped 17 previous similar messages
Mar 31 16:37:06 atlas-tds-mds1 kernel: [ 8894.611888] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 16:37:06 atlas-tds-mds1 kernel: [ 8894.645071] LustreError: Skipped 4 previous similar messages
Mar 31 16:41:03 atlas-tds-mds1 kernel: [ 9132.252761] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 16:41:03 atlas-tds-mds1 kernel: [ 9132.258815] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 16:41:03 atlas-tds-mds1 kernel: [ 9132.258818] Lustre: Skipped 16 previous similar messages
Mar 31 16:41:03 atlas-tds-mds1 kernel: [ 9132.343741] Lustre: Skipped 19 previous similar messages
Mar 31 16:49:36 atlas-tds-mds1 kernel: [ 9644.862649] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 16:49:36 atlas-tds-mds1 kernel: [ 9644.895158] LustreError: Skipped 4 previous similar messages
Mar 31 16:51:03 atlas-tds-mds1 kernel: [ 9732.474466] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 16:51:03 atlas-tds-mds1 kernel: [ 9732.497786] Lustre: Skipped 13 previous similar messages
Mar 31 16:51:03 atlas-tds-mds1 kernel: [ 9732.504038] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 16:51:03 atlas-tds-mds1 kernel: [ 9732.504041] Lustre: Skipped 16 previous similar messages
Mar 31 17:01:03 atlas-tds-mds1 kernel: [10332.694093] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 17:01:03 atlas-tds-mds1 kernel: [10332.721456] Lustre: Skipped 13 previous similar messages
Mar 31 17:01:03 atlas-tds-mds1 kernel: [10332.726024] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:01:03 atlas-tds-mds1 kernel: [10332.726027] Lustre: Skipped 15 previous similar messages
Mar 31 17:02:06 atlas-tds-mds1 kernel: [10395.135866] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 17:02:06 atlas-tds-mds1 kernel: [10395.174382] LustreError: Skipped 4 previous similar messages
Mar 31 17:11:03 atlas-tds-mds1 kernel: [10932.921709] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) reconnecting
Mar 31 17:11:03 atlas-tds-mds1 kernel: [10932.945031] Lustre: Skipped 13 previous similar messages
Mar 31 17:11:03 atlas-tds-mds1 kernel: [10932.955958] Lustre: atlastds-MDT0000: Client 411700ef-c5bd-ec9f-cb66-dc45aad45f27 (at 28@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:11:03 atlas-tds-mds1 kernel: [10933.005431] Lustre: Skipped 17 previous similar messages
Mar 31 17:14:36 atlas-tds-mds1 kernel: [11145.408358] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 17:14:36 atlas-tds-mds1 kernel: [11145.444286] LustreError: Skipped 4 previous similar messages
Mar 31 17:21:03 atlas-tds-mds1 kernel: [11533.185061] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 17:21:03 atlas-tds-mds1 kernel: [11533.209210] Lustre: Skipped 17 previous similar messages
Mar 31 17:21:03 atlas-tds-mds1 kernel: [11533.228256] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:21:03 atlas-tds-mds1 kernel: [11533.269410] Lustre: Skipped 15 previous similar messages
Mar 31 17:24:36 atlas-tds-mds1 kernel: [11745.644028] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 17:24:36 atlas-tds-mds1 kernel: [11745.678038] LustreError: Skipped 3 previous similar messages
Mar 31 17:33:33 atlas-tds-mds1 kernel: [12283.399181] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 17:33:33 atlas-tds-mds1 kernel: [12283.404774] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:33:33 atlas-tds-mds1 kernel: [12283.404777] Lustre: Skipped 16 previous similar messages
Mar 31 17:33:33 atlas-tds-mds1 kernel: [12283.488666] Lustre: Skipped 19 previous similar messages
Mar 31 17:37:06 atlas-tds-mds1 kernel: [12495.922545] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 17:37:06 atlas-tds-mds1 kernel: [12495.957525] LustreError: Skipped 4 previous similar messages
Mar 31 17:43:33 atlas-tds-mds1 kernel: [12883.623603] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 17:43:33 atlas-tds-mds1 kernel: [12883.638518] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:43:33 atlas-tds-mds1 kernel: [12883.638521] Lustre: Skipped 15 previous similar messages
Mar 31 17:43:33 atlas-tds-mds1 kernel: [12883.712116] Lustre: Skipped 15 previous similar messages
Mar 31 17:49:36 atlas-tds-mds1 kernel: [13246.172990] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 17:49:36 atlas-tds-mds1 kernel: [13246.207228] LustreError: Skipped 4 previous similar messages
Mar 31 17:53:33 atlas-tds-mds1 kernel: [13483.845666] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 17:53:33 atlas-tds-mds1 kernel: [13483.876256] Lustre: Skipped 12 previous similar messages
Mar 31 17:53:33 atlas-tds-mds1 kernel: [13483.888378] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 17:53:33 atlas-tds-mds1 kernel: [13483.888381] Lustre: Skipped 15 previous similar messages
Mar 31 18:02:06 atlas-tds-mds1 kernel: [13996.445823] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 18:02:06 atlas-tds-mds1 kernel: [13996.478431] LustreError: Skipped 4 previous similar messages
Mar 31 18:03:33 atlas-tds-mds1 kernel: [14084.081631] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 18:03:33 atlas-tds-mds1 kernel: [14084.109743] Lustre: Skipped 14 previous similar messages
Mar 31 18:03:33 atlas-tds-mds1 kernel: [14084.120624] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:03:33 atlas-tds-mds1 kernel: [14084.169960] Lustre: Skipped 18 previous similar messages
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.813034] LustreError: 12440:0:(ldlm_lock.c:2149:ldlm_lock_cancel()) ### lock still has references ns: mdt-atlastds-MDT0000_UUID lock: ffff880fe9d09d80/0xcfedff8c85b227aa lrc: 6/1,0 mode: CR/CR res: [0x200004281:0x1:0x0].0 bits 0x9 rrc: 5 type: IBT flags: 0x200000000020 nid: 29@gni2 remote: 0x46fe20a418cdb11b expref: 10 pid: 13062 timeout: 4309419077 lvb_type: 0 used 0
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.916752] LustreError: 12440:0:(ldlm_lock.c:2150:ldlm_lock_cancel()) LBUG
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.935737] Pid: 12440, comm: mdt01_118
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.946230] 
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.946231] Call Trace:
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.957063]  [<ffffffffa03dd895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14609.985613]  [<ffffffffa03dde97>] lbug_with_loc+0x47/0xb0 [libcfs]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.005588]  [<ffffffffa06c44dd>] ldlm_lock_cancel+0x1dd/0x1e0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.026007]  [<ffffffffa06e892e>] ldlm_handle_ast_error+0x1de/0x540 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.046645]  [<ffffffffa06fdefc>] ? after_reply+0x7fc/0xd90 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.066794]  [<ffffffffa06e8e5e>] ldlm_cb_interpret+0x1ce/0x2b0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.095472]  [<ffffffffa070206c>] ptlrpc_check_set+0x2ac/0x1b20 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.115833]  [<ffffffff810813d0>] ? process_timeout+0x0/0x10
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.135682]  [<ffffffffa0703c0a>] ptlrpc_set_wait+0x32a/0x8c0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.155960]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.176033]  [<ffffffffa06c2e20>] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.196539]  [<ffffffffa06c5f8b>] ldlm_run_ast_work+0x1db/0x490 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.225323]  [<ffffffffa06f39e7>] ldlm_process_inodebits_lock+0xf7/0x3e0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.246228]  [<ffffffffa06c58e5>] ldlm_lock_enqueue+0x405/0x8d0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.266407]  [<ffffffffa06e46db>] ldlm_cli_enqueue_local+0x18b/0x5d0 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.295502]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.315884]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.336097]  [<ffffffffa0dbad6d>] mdt_object_lock0+0x33d/0xaf0 [mdt]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.356431]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.376657]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 18:12:19 atlas-tds-mds1 kernel: [14610.405537]  [<ffffffffa0dbb5e4>] mdt_object_lock+0x14/0x20 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.425576]  [<ffffffffa0dd434e>] mdt_reint_unlink+0x66e/0x1010 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.445775]  [<ffffffffa0dd0d21>] mdt_reint_rec+0x41/0xe0 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.465751]  [<ffffffffa0db5b73>] mdt_reint_internal+0x4c3/0x780 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.486029]  [<ffffffffa0db5e74>] mdt_reint+0x44/0xe0 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.505965]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.526021]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.546998]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.575416]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.596701]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.616222]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.636715]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.665637]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.676891]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.696888]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.725492]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.736733]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.756918]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.776913]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.796585] 
Mar 31 18:12:20 atlas-tds-mds1 kernel: [14610.805687] LustreError: dumping log to /tmp/lustre-log.1396303940.12440
Mar 31 18:13:33 atlas-tds-mds1 kernel: [14684.308004] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 18:13:33 atlas-tds-mds1 kernel: [14684.333503] Lustre: Skipped 16 previous similar messages
Mar 31 18:13:33 atlas-tds-mds1 kernel: [14684.344474] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:13:33 atlas-tds-mds1 kernel: [14684.393877] Lustre: Skipped 15 previous similar messages
Mar 31 18:14:36 atlas-tds-mds1 kernel: [14746.719262] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 18:14:36 atlas-tds-mds1 kernel: [14746.756575] LustreError: Skipped 4 previous similar messages
Mar 31 18:14:50 atlas-tds-mds1 kernel: [14760.791071] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 3@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff880fe9509480/0xcfedff8c85b227b8 lrc: 6/1,0 mode: CR/CR res: [0x200004281:0x1:0x0].0 bits 0x9 rrc: 5 type: IBT flags: 0x200000000020 nid: 3@gni2 remote: 0x1c5d81965f87495 expref: 10 pid: 13083 timeout: 4309419077 lvb_type: 0 used 0
Mar 31 18:14:50 atlas-tds-mds1 kernel: [14760.913494] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 29@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff880fe9d09d80/0xcfedff8c85b227aa lrc: 6/1,0 mode: CR/CR res: [0x200004281:0x1:0x0].0 bits 0x9 rrc: 5 type: IBT flags: 0x200000000020 nid: 29@gni2 remote: 0x46fe20a418cdb11b expref: 10 pid: 13062 timeout: 4309419077 lvb_type: 0 used 0
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.346130] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.361092] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.631336] CPU 9 
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.632349] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.902660] 
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.911821] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.943016] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.972622] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14840.991772] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.012336] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.041319] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.061874] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.082556] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.111491] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.133021] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.152445] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.181376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.202146] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.222865] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.252871] Stack:
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.261919]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.282654] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.311985] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.341435] Call Trace:
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.343303]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.371474]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.391914]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.421896]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.442485]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.471907]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.492964]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.522063]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.543052]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.572071]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.592071]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.612478]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.632805]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.661894]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.681981]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.702483]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.721956]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.742376]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.762668]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.782163] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.843234] Call Trace:
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.852542]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.872933]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.901632]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.923161]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.952201]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14841.973510]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.002604]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.023446]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.052323]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.073176]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.092958]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.121735]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.142321]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.162820]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.183015]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.211721]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.233549]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.251943]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:16:11 atlas-tds-mds1 kernel: [14842.272579]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.376881] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.392383] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.662898] CPU 9 
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.663915] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.942545] 
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.943816] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14924.982792] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.004092] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.023392] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.044057] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.073062] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.093752] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.122681] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.143466] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.173466] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.192902] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.213423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.242671] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.263278] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.293491] Stack:
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.295029]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.323159] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.344164] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.373888] Call Trace:
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.383009]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.403526]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.423740]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.453538]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.474456]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.503744]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.533201]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.553985]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.583246]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.603981]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.623949]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.644363]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.673301]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.693820]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.713990]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.734321]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.753850]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.774344]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.802922]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.814207] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.883327] Call Trace:
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.892942]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.913387]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.933925]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.963741]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14925.984456]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.013933]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.043356]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.064203]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.093387]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.114249]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.134153]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.154373]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.183550]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.203885]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.224050]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.244338]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.263857]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.284239]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:17:35 atlas-tds-mds1 kernel: [14926.313090]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.407621] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.423739] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.694185] CPU 9 
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.695257] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.965003] 
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15008.974053] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.005529] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.035040] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.054332] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.075036] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.103973] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.124533] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.145402] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.174224] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.195726] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.215122] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.244198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.264947] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.285445] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.315835] Stack:
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.324848]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.345505] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.374986] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.404881] Call Trace:
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.414192]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.434518]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.454700]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.484473]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.505274]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.534703]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.555969]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.585077]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.606074]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.645591]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.664387]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.684833]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.705330]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.734174]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.754233]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.774806]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.794389]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.814646]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.834996]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.854615] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.915647] Call Trace:
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.925005]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.945457]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.965699]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15009.995600]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.024713]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.045798]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.075117]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.095856]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.124938]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.145873]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.165783]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.194419]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.215001]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.235498]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.255517]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.275881]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.295329]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.315531]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:18:59 atlas-tds-mds1 kernel: [15010.335934]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15092.438349] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15092.455017] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15092.725582] CPU 9 
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15092.726593] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15092.996668] 
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.005809] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.037150] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.066610] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.085875] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.106551] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.135514] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.155930] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.176569] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.205700] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.227173] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.246600] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.275729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.296307] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.316759] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.347040] Stack:
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.356147]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.376797] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.406338] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.435970] Call Trace:
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.437830]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.465633]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.486048]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.515805]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.536601]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.566082]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.587418]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.616243]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.636996]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.666263]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.686087]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.706650]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.735557]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.755927]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.776145]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.796448]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.815902]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.836490]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.856737]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.876318] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.937313] Call Trace:
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.946772]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.966983]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15093.995667]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.017100]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.046303]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.067332]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.096653]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.117451]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.146671]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.167446]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.187293]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.215885]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.236507]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.256834]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.276998]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.297321]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.316868]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.337053]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:20:23 atlas-tds-mds1 kernel: [15094.365941]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15176.469065] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15176.486350] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15176.757221] CPU 9 
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15176.766057] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.028040] 
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.037352] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.076780] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.108577] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.117723] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.146831] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.167411] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.188068] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.217230] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.237764] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.267821] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.287438] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.307992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.337175] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.357868] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.387872] Stack:
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.396872]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.417642] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.446725] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.468053] Call Trace:
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.477486]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.497859]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.518234]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.548108]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.577185]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.598147]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.627567]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.648355]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.677584]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.698536]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.718492]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.747268]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.767636]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.788212]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.808325]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.836901]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.848052]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.876935]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.897304]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.916928] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.977883] Call Trace:
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15177.987141]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.007559]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.028072]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.057760]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.078624]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.108148]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.137558]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.158354]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.187402]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.208157]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.227972]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.248290]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.277289]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.297650]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.317792]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.338153]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.357737]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.378061]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:21:47 atlas-tds-mds1 kernel: [15178.398360]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15260.499770] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15260.517699] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15260.787988] CPU 9 
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15260.788990] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.058965] 
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.068035] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.099352] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.128878] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.148086] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.168860] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.189613] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.218487] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.239055] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.268024] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.289615] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.309170] CR2: 00007f9c44316000 CR3: 00000010747a3000 CR4: 00000000001407e0
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.338021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.358893] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.379316] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.409370] Stack:
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.418408]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.439416] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.468392] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.490021] Call Trace:
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.499169]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.528089]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.548308]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.569855]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.599203]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.619951]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.649433]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.678532]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.699516]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.728544]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.748363]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.768846]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.789543]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.818171]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.838413]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.858787]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.878223]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.898625]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.919124]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.938614] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15261.999699] Call Trace:
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.009068]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.029677]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.058275]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.079834]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.109049]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.130021]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.159439]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.180204]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.209617]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.230230]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.258733]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.289675]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.308821]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.329298]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.349534]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.369817]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.389325]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.409817]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:23:11 atlas-tds-mds1 kernel: [15262.438427]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:23:34 atlas-tds-mds1 kernel: [15284.605248] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 18:23:34 atlas-tds-mds1 kernel: [15284.624534] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:23:34 atlas-tds-mds1 kernel: [15284.624537] Lustre: Skipped 10 previous similar messages
Mar 31 18:23:34 atlas-tds-mds1 kernel: [15284.687985] Lustre: Skipped 11 previous similar messages
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15344.530463] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15344.549012] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15344.819703] CPU 9 
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15344.820698] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.099393] 
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.100624] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.139849] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.161114] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.180398] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.209471] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.230066] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.250692] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.279321] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.299935] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.330192] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.349484] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.370105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.390772] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.419716] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.449724] Stack:
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.451268]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.479546] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.500441] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.530171] Call Trace:
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.539429]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.559991]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.580252]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.609987]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.631039]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.660433]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.689822]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.710594]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.739911]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.760720]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.780658]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.801096]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.829907]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.850431]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.870785]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.891266]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.910487]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.930965]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.959729]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15345.970960] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.040143] Call Trace:
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.049575]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.070000]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.090359]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.120388]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.141463]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.170609]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.200067]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.220648]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.250034]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.270892]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.290836]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.310996]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.340133]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.360731]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.381046]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.401144]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.420860]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.449733]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:24:35 atlas-tds-mds1 kernel: [15346.470136]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:24:36 atlas-tds-mds1 kernel: [15346.956202] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 18:24:36 atlas-tds-mds1 kernel: [15346.990343] LustreError: Skipped 3 previous similar messages
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15428.561144] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15428.580337] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15428.841852] CPU 9 
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15428.850817] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.121619] 
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.130821] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.162143] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.191522] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.210806] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.231334] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.252096] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.281195] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.301911] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.330857] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.352182] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.371755] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.400648] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.421253] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.441922] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.472098] Stack:
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.481137]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.502213] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.531268] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.560704] Call Trace:
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.562570]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.601225]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.620977]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.642547]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.671730]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.701154]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.722103]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.751167]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.772196]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.801403]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.821400]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.841777]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.862268]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.891055]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.911222]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.931624]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.951323]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.971619]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15429.992166]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.011563] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.072597] Call Trace:
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.081861]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.102346]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.131169]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.152419]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.181811]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.202754]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.232200]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.252638]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.282260]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.302691]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.331029]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.351422]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.372000]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.392443]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.412489]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.441438]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.452645]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.481472]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:25:59 atlas-tds-mds1 kernel: [15430.501916]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:26:50 atlas-tds-mds1 kernel: [15481.312480] Lustre: 12536:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply
Mar 31 18:26:50 atlas-tds-mds1 kernel: [15481.312481]   req@ffff88075b938800 x1463855620041648/t0(0) o36->2892c2bc-6435-ceee-96aa-67a7e6401f59@12@gni2:0/0 lens 496/3128 e 3 to 0 dl 1396304815 ref 2 fl Interpret:/0/0 rc 0/0
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15512.591815] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15512.611702] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15512.882168] CPU 9 
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15512.883321] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.153218] 
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.162338] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.193639] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.223389] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.242644] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.263276] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.292119] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.312750] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.333544] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.362583] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.392627] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.412001] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.432692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.453267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.482194] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.503823] Stack:
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.512945]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.533780] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.562836] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.592611] Call Trace:
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.594619]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.622066]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.642487]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.663995]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.692885]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.714031]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.743444]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.764107]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.793194]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.822531]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.842339]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.862816]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.883299]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.912185]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.932381]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.952795]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.972438]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15513.992790]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.013004]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.032573] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.093606] Call Trace:
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.102833]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.123225]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.143715]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.173561]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.202690]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.223666]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.253148]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.273974]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.303137]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.323980]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.352346]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.372816]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.393339]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.413637]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.433824]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.462587]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.473900]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.502410]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:27:23 atlas-tds-mds1 kernel: [15514.522603]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15596.622476] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15596.643061] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15596.914091] CPU 9 
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15596.922995] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.193429] 
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.194657] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.233439] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.255024] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.274181] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.303245] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.323830] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.344530] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.373473] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.394081] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.424086] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.443506] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.464226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.484733] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.513890] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.543928] Stack:
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.545472]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.573729] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.594624] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.624503] Call Trace:
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.633647]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.654062]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.674422]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.704456]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.725393]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.754412]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.783875]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.804454]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.833858]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.854420]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.874622]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.894992]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.923680]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.944134]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.964361]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15597.984892]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.004351]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.024654]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.053553]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.064800] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.134154] Call Trace:
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.143773]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.164145]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.184623]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.214353]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.235343]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.264601]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.294036]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.314781]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.344059]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.364848]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.384780]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.405106]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.433905]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.454444]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.474459]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.494947]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.514291]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.534689]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:28:47 atlas-tds-mds1 kernel: [15598.554974]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15680.653126] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15680.665324] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15680.935542] CPU 9 
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15680.944435] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.215122] 
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.216335] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.255577] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.285098] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.296130] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.325159] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.345872] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.374631] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.395265] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.415801] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.445788] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.465215] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.485894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.514857] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.535639] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.565464] Stack:
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.566998]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.595118] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.616276] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.646227] Call Trace:
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.655294]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.675674]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.696097]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.725758]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.746418]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.775968]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.805346]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.826275]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.855385]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.876276]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.896081]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.924902]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.945440]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.965779]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15681.985947]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.006313]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.025802]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.046004]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.074885]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.086098] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.165794] Call Trace:
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.167671]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.194962]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.215310]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.236758]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.265949]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.295327]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.316460]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.345512]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.366472]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.395385]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.415411]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.435739]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.456237]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.485035]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.505038]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.525626]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.545094]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.565421]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:30:11 atlas-tds-mds1 kernel: [15682.585865]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15764.683765] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15764.696197] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15764.966363] CPU 9 
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15764.967363] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.237477] 
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.246626] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.286462] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.307698] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.327002] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.347402] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.376490] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.397097] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.426224] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.446794] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.476373] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.487536] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.516377] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.537075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.566181] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.587791] Stack:
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.596904]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.626014] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.646762] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.676622] Call Trace:
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.678350]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.706255]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.726666]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.756455]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.777227]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.806647]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.827903]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.856855]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.877765]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.906874]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.926911]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.947390]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.976384]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15765.996732]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.016771]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.037089]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.056663]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.077006]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.097294]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.116946] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.177915] Call Trace:
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.187387]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.207707]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.236287]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.257874]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.287034]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.307976]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.337292]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.357941]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.387320]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.407806]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.428026]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.456614]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.477163]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.497656]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.517869]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.546698]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.557820]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.586546]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:31:35 atlas-tds-mds1 kernel: [15766.606948]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.251428] LNet: Service thread pid 12440 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.304045] LNet: Skipped 1 previous similar message
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.322649] Pid: 12440, comm: mdt01_118
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.333072] 
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.333072] Call Trace:
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.343730]  [<ffffffff8150c388>] ? schedule+0x178/0x3b2
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.363484]  [<ffffffffa03ddecd>] lbug_with_loc+0x7d/0xb0 [libcfs]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.383396]  [<ffffffffa06c44dd>] ldlm_lock_cancel+0x1dd/0x1e0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.403946]  [<ffffffffa06e892e>] ldlm_handle_ast_error+0x1de/0x540 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.432771]  [<ffffffffa06fdefc>] ? after_reply+0x7fc/0xd90 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.452969]  [<ffffffffa06e8e5e>] ldlm_cb_interpret+0x1ce/0x2b0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.473199]  [<ffffffffa070206c>] ptlrpc_check_set+0x2ac/0x1b20 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.493687]  [<ffffffff810813d0>] ? process_timeout+0x0/0x10
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.513613]  [<ffffffffa0703c0a>] ptlrpc_set_wait+0x32a/0x8c0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.533677]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.553927]  [<ffffffffa06c2e20>] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.574390]  [<ffffffffa06c5f8b>] ldlm_run_ast_work+0x1db/0x490 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.603037]  [<ffffffffa06f39e7>] ldlm_process_inodebits_lock+0xf7/0x3e0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.624325]  [<ffffffffa06c58e5>] ldlm_lock_enqueue+0x405/0x8d0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.644096]  [<ffffffffa06e46db>] ldlm_cli_enqueue_local+0x18b/0x5d0 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.673247]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.693557]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.713795]  [<ffffffffa0dbad6d>] mdt_object_lock0+0x33d/0xaf0 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.734282]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.754328]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.783298]  [<ffffffffa0dbb5e4>] mdt_object_lock+0x14/0x20 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.803439]  [<ffffffffa0dd434e>] mdt_reint_unlink+0x66e/0x1010 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.823687]  [<ffffffffa0dd0d21>] mdt_reint_rec+0x41/0xe0 [mdt]
Mar 31 18:32:19 atlas-tds-mds1 kernel: [15810.843866]  [<ffffffffa0db5b73>] mdt_reint_internal+0x4c3/0x780 [mdt]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.864011]  [<ffffffffa0db5e74>] mdt_reint+0x44/0xe0 [mdt]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.883829]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.903925]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.933325]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.953354]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.974623]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15810.994379]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.023168]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.043713]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.063203]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.093875]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.112991]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.124370]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.144304]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.164425]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.183925] 
Mar 31 18:32:20 atlas-tds-mds1 kernel: [15811.192970] LustreError: dumping log to /tmp/lustre-log.1396305140.12440
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15848.714395] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15848.726986] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15848.997698] CPU 9 
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15848.998838] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.277589] 
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.278814] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.317848] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.347617] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.358646] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.387648] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.408227] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.428701] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.457700] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.478229] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.508110] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.527492] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.548295] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.568606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.597561] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.627699] Stack:
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.629100]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.657553] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.678607] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.708558] Call Trace:
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.717597]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.737983]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.758381]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.788363]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.809222]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.838521]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.867826]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.888589]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.917835]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.938790]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.958723]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15849.979076]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.007923]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.028570]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.048613]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.069075]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.088307]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.108871]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.137554]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.148966] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.218203] Call Trace:
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.227803]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.248208]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.268548]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.298574]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.319168]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.348634]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.378037]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.398815]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.428010]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.448829]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.468731]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.488983]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.518085]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.538427]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.558487]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.578999]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.598651]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.619007]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:32:59 atlas-tds-mds1 kernel: [15850.639135]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15932.745015] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15932.758484] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.028817] CPU 9 
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.029904] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.299467] 
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.308543] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.339844] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.369319] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.388552] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.409356] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.429877] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.458673] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.479392] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.499996] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.529623] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.549192] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.569827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.598693] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.619278] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.649241] Stack:
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.650930]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.679291] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.700273] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.730441] Call Trace:
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.739499]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.759880]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.779994]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.809878]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.830730]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.860099]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.889385]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.920914]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.939972]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.960793]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15933.989027]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.009405]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.029952]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.050256]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.078852]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.099007]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.110201]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.139026]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.159421]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.178869] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.239628] Call Trace:
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.249244]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.269505]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.290010]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.319890]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.340606]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.370088]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.399542]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.420147]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.449349]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.470281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.490086]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.510568]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.539602]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.560121]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.580212]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.600733]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.619883]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.640470]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:34:23 atlas-tds-mds1 kernel: [15934.669245]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:35:03 atlas-tds-mds1 kernel: [15974.364542] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 18:35:03 atlas-tds-mds1 kernel: [15974.394654] Lustre: Skipped 14 previous similar messages
Mar 31 18:35:03 atlas-tds-mds1 kernel: [15974.413866] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:35:03 atlas-tds-mds1 kernel: [15974.454891] Lustre: Skipped 15 previous similar messages
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16016.775625] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16016.789839] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.060128] CPU 9 
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.061139] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.331113] 
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.340077] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.371507] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.401036] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.420470] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.440953] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.461508] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.490515] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.511037] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.539990] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.561499] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.580960] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.610033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.630592] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.651269] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.681504] Stack:
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.690740]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.711272] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.740419] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.762002] Call Trace:
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.771163]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.791509]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.820071]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.841572]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.870669]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.891584]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.920868]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.941850]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.971129]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16017.991937]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.020313]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.040569]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.061061]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.081574]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.101608]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.130185]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.141474]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.170214]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.190566]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.201627] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.271270] Call Trace:
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.280670]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.301051]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.321531]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.351352]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.372118]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.401436]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.430885]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.451429]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.480904]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.501459]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.521608]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.550372]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.570986]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.591462]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.611812]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.631894]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.651589]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.680415]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:35:47 atlas-tds-mds1 kernel: [16018.700748]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:37:06 atlas-tds-mds1 kernel: [16097.235606] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 18:37:06 atlas-tds-mds1 kernel: [16097.270065] LustreError: Skipped 4 previous similar messages
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16100.806225] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16100.820999] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.091308] CPU 9 
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.092347] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.371774] 
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.373035] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.412021] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.441624] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.452535] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.481437] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.502144] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.522564] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.551863] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.572372] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.602266] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.621689] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.642297] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.671503] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.692110] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.722489] Stack:
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.731362]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.752297] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.781371] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.803020] Call Trace:
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.812129]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.832703]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.852846]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.882742]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.911996]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.933107]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.962463]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16101.983326]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.012440]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.033068]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.061474]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.081668]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.102266]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.122735]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.142939]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.171487]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.182786]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.211716]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.232096]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.251536] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.312498] Call Trace:
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.321974]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.342541]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.362860]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.392579]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.413633]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.442816]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.472232]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.492969]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.522223]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.542968]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.562905]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.583130]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.612262]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.632780]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.652819]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.673127]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.692499]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.712905]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:37:11 atlas-tds-mds1 kernel: [16102.741724]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16184.836817] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16184.852513] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.122878] CPU 9 
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.123955] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.393637] 
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.402725] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.433834] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.463661] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.482741] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.503164] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.523842] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.552798] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.573381] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.593877] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.623987] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.643233] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.663916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.692869] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.713416] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.743668] Stack:
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.752715]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.773219] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.802664] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.824149] Call Trace:
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.833280]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.853674]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.874139]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.903926]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.924597]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.954118]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16185.983541]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.004289]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.033560]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.054450]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.074238]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.103018]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.124160]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.144200]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.164152]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.193061]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.204312]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.233059]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.253452]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.273083] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.334129] Call Trace:
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.343492]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.363828]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.384027]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.413889]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.445128]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.464265]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.493723]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.514459]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.543780]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.564486]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.584489]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.612999]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.633396]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.653965]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.674157]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.694611]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.713989]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.734298]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:38:35 atlas-tds-mds1 kernel: [16186.763264]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16268.867399] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16268.883861] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.145389] CPU 9 
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.154310] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.425084] 
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.434097] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.465309] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.494997] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.514283] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.534825] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.564092] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.584615] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.605389] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.634098] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.655643] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.675208] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.704145] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.724742] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.745398] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.775693] Stack:
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.784729]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.805285] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.834553] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.864008] Call Trace:
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.865896]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.894283]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.914617]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.944427]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.965210]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16269.994813]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.015889]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.044837]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.065755]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.094848]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.114885]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.135116]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.155639]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.184489]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.204727]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.225270]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.244713]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.264963]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.285489]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.304756] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.365904] Call Trace:
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.375301]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.395786]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.424265]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.446106]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.475086]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.496254]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.525471]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.546089]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.575458]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.604689]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.624697]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.645054]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.665644]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.694317]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.714538]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.734971]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.754512]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.774941]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:39:59 atlas-tds-mds1 kernel: [16270.795480]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16352.897972] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16352.914956] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.185311] CPU 9 
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.186476] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.465146] 
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.466439] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.505584] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.526839] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.546335] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.575273] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.595864] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.616549] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.645311] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.666252] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.695852] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.715486] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.736028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.756714] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.785845] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.815713] Stack:
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.817257]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.845494] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.866558] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.896657] Call Trace:
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.905987]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.926627]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.947010]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.976509]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16353.997313]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.026563]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.056013]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.076744]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.106033]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.126901]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.146895]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.175529]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.196299]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.216779]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.236597]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.256972]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.276670]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.296808]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.325732]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.336962] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.405935] Call Trace:
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.415553]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.435902]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.456321]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.485941]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.506898]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.536429]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.557127]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.586225]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.607375]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.636253]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.656310]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.676476]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.696988]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.725874]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.746023]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.766547]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.786036]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.806281]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:41:23 atlas-tds-mds1 kernel: [16354.826798]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16436.928537] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16436.946493] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.216763] CPU 9 
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.217773] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.487571] 
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.496561] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.527790] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.557526] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.576738] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.597147] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.617991] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.646777] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.667369] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.688028] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.718112] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.737367] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.758201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.787027] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.807747] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.837844] Stack:
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.846934]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.867672] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.896854] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.918197] Call Trace:
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.927781]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.948183]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.976755]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16437.998192]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.027522]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.048448]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.077922]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.098468]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.127813]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.148465]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.176816]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.197033]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.217610]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.238052]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.258311]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.287163]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.298266]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.327082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.347216]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.366846] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.427891] Call Trace:
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.437254]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.457851]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.478226]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.508048]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.528712]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.558257]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.587719]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.608441]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.637751]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.658611]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.678401]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.707226]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.727933]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.748245]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.768499]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.797205]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.808324]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.837210]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:42:47 atlas-tds-mds1 kernel: [16438.857475]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16520.959162] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16520.977662] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.248111] CPU 9 
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.249181] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.518941] 
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.527819] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.559146] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.588820] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.608088] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.628667] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.649423] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.678088] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.698912] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.719604] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.749481] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.768923] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.797918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.818551] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.839081] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.869463] Stack:
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.878548]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.899241] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.928603] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.957989] Call Trace:
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.959819]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16521.988343]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.008430]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.038405]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.059865]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.088830]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.109746]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.138876]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.159862]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.189119]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.209029]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.229425]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.258538]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.279257]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.299256]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.319797]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.339275]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.359651]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.388362]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.399440] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.468597] Call Trace:
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.478207]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.498407]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.518658]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.540047]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.569386]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.598864]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.619770]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.648879]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.669679]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.698840]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.718981]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.739103]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.759868]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.788491]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.808729]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.829129]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.848757]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.869145]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:44:11 atlas-tds-mds1 kernel: [16522.889615]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:45:03 atlas-tds-mds1 kernel: [16574.585599] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 18:45:03 atlas-tds-mds1 kernel: [16574.608402] Lustre: Skipped 15 previous similar messages
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16604.990416] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.008985] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.279415] CPU 9 
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.280575] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.550609] 
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.559724] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.599407] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.620854] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.639959] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.660623] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.689427] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.710187] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.730721] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.759934] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.789647] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.800705] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.829980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.850517] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.879440] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.901101] Stack:
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.910249]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.939303] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.960227] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.990122] Call Trace:
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16605.991966]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.019531]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.040037]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.070036]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.090535]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.120063]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.141063]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.170195]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.191216]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.220395]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.240184]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.260849]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.289678]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.310222]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.330208]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.350603]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.369889]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.390247]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.410863]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.430299] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.491321] Call Trace:
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.500652]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.520956]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.549569]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.571036]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.600380]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.621654]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.650903]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.680146]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.700929]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.730192]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.750009]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.770682]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.791248]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.820235]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.840254]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.860481]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.880107]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.900511]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:45:35 atlas-tds-mds1 kernel: [16606.920654]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:46:03 atlas-tds-mds1 kernel: [16634.988789] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:46:03 atlas-tds-mds1 kernel: [16635.021701] Lustre: Skipped 17 previous similar messages
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.021656] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.040286] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.310599] CPU 9 
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.311604] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.581459] 
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.590498] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.621651] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.651466] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.670506] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.691125] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.712048] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.740959] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.761647] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.790562] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.812224] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.831861] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.860837] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.881392] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.901880] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.931971] Stack:
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.941043]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.961942] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16689.991008] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.012680] Call Trace:
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.021625]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.042003]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.070717]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.092403]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.121545]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.142556]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.172053]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.201102]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.222103]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.242580]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.271144]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.291541]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.312082]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.332385]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.360781]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.381014]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.392232]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.421061]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.441664]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.460868] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.521882] Call Trace:
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.531315]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.551691]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.572033]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.601884]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.622800]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.652241]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.681507]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.702414]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.731749]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.752434]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.772362]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.800977]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.821599]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.842087]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.862325]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.882488]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.902123]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.922603]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:46:59 atlas-tds-mds1 kernel: [16690.951235]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.052881] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.071602] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.341963] CPU 9 
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.343128] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.612911] 
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.622030] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.653279] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.683001] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.702247] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.722788] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.743311] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.772298] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.793120] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.821947] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.853927] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.863351] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.892164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.912889] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.933455] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.963426] Stack:
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.972557]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16773.993231] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.022558] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.051948] Call Trace:
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.053728]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.082213]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.102402]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.123845]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.153184]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.182464]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.203557]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.232494]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.253370]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.282459]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.302552]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.322900]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.343470]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.372357]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.392573]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.412919]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.432543]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.452924]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.473289]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.493143] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.553978] Call Trace:
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.563303]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.583646]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.612621]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.634207]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.663238]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.692639]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.713869]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.742864]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.763762]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.793184]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.813092]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.833443]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.853884]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.882713]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.902878]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.923071]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.942742]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.962947]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:48:23 atlas-tds-mds1 kernel: [16774.983306]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:49:36 atlas-tds-mds1 kernel: [16847.487886] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 18:49:36 atlas-tds-mds1 kernel: [16847.520516] LustreError: Skipped 4 previous similar messages
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.084092] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.102985] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.373447] CPU 9 
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.374602] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.644458] 
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.653365] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.684611] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.714336] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.733551] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.754102] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.774639] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.803781] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.824501] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.853447] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.874942] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.894509] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.923646] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.944226] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.964707] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16857.994785] Stack:
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.003856]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.024722] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.053860] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.083309] Call Trace:
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.085078]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.113381]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.133790]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.155309]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.184194]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.205233]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.234513]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.263779]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.284733]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.305422]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.333612]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.354019]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.374625]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.403441]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.423683]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.444024]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.463679]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.484246]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.504622]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.524110] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.585248] Call Trace:
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.594523]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.614892]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.643527]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.665032]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.694117]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.715211]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.744589]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.765332]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.794582]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.815378]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.843635]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.864161]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.884613]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.905061]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.925140]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.953806]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.964923]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16858.995604]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:49:47 atlas-tds-mds1 kernel: [16859.014045]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.115290] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.134430] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.404727] CPU 9 
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.405882] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.676114] 
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.685039] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.724486] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.745969] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.765403] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.785881] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.814824] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.835717] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.864591] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.885390] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.914929] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.934573] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.954962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16941.975775] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.004766] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.026787] Stack:
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.035586]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.064589] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.085554] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.115201] Call Trace:
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.117087]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.144868]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.165241]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.195086]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.215964]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.245503]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.266372]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.295513]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.316564]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.345811]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.365736]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.385988]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.414769]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.435364]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.455752]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.476177]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.495727]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.516221]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.544825]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.555973] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.625037] Call Trace:
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.626963]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.655065]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.675428]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.705442]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.726200]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.755665]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.776686]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.805758]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.826689]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.855758]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.875740]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.896074]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.925163]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.945558]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.965712]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16942.986118]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16943.005721]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16943.026251]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:51:11 atlas-tds-mds1 kernel: [16943.046686]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.146473] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.165791] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.436089] CPU 9 
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.437092] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.707352] 
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.716410] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.747450] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.777169] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.796384] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.816933] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.845840] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.866440] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.887116] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.915918] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.937618] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.956982] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17025.985954] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.006638] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.027381] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.057428] Stack:
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.066337]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.087128] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.116392] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.137967] Call Trace:
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.147123]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.176016]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.196393]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.217693]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.246953]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.268043]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.297526]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.326642]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.347564]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.387147]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.406281]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.426684]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.447229]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.467602]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.487681]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.516303]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.527677]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.556403]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.576794]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.596355] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.657284] Call Trace:
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.666578]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.687150]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.707651]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.737617]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.758465]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.787576]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.817009]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.837731]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.866652]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.887403]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.907573]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.927907]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.956870]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.977406]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17026.997404]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17027.017798]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17027.037439]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17027.057761]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:52:35 atlas-tds-mds1 kernel: [17027.086552]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.177644] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.197075] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.467210] CPU 9 
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.468292] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.738060] 
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.747174] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.778301] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.808097] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.827351] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.848275] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.877239] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.897835] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.918536] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.947404] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.969161] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17109.988477] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.017629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.038345] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.058771] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.088911] Stack:
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.097843]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.118681] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.147821] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.177287] Call Trace:
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.179164]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.207314]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.227750]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.249077]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.278584]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.307963]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.329066]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.358128]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.379090]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.408328]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.428178]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.448696]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.477624]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.498197]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.518414]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.538577]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.558274]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.578581]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.598888]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.618383] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.687485] Call Trace:
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.689330]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.717504]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.737923]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.759480]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.788552]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.818079]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.839190]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.868103]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.888903]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.918197]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.938211]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.958394]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17110.979113]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.007969]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.028188]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.048573]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.068243]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.088431]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:53:59 atlas-tds-mds1 kernel: [17111.108887]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.208802] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.228440] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.498569] CPU 9 
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.499734] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.769565] 
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.778629] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.809776] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.849762] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.859598] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.888694] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.909467] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.930101] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.958938] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17193.979730] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.009569] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.029097] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.049827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.078569] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.099192] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.129375] Stack:
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.130767]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.159183] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.188622] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.210310] Call Trace:
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.219439]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.239804]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.260252]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.289982]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.319261]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.340197]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.369657]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.390405]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.419677]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.440558]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.468716]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.489153]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.509697]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.530025]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.550042]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.578751]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.589956]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.618838]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.639443]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.659006] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.719941] Call Trace:
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.729381]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.749655]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.769893]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.799844]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.820506]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.850261]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.879423]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.900082]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.929274]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.950078]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.970041]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17194.990388]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.019389]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.039618]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.059745]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.080091]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.099765]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.119997]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:55:23 atlas-tds-mds1 kernel: [17195.148976]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:56:03 atlas-tds-mds1 kernel: [17235.221051] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 18:56:03 atlas-tds-mds1 kernel: [17235.244520] Lustre: Skipped 16 previous similar messages
Mar 31 18:56:03 atlas-tds-mds1 kernel: [17235.263921] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 18:56:03 atlas-tds-mds1 kernel: [17235.304965] Lustre: Skipped 14 previous similar messages
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.239947] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.259567] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.529885] CPU 9 
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.530989] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.800862] 
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.809769] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.841229] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.870544] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.889836] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.910573] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.931313] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.960185] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17277.981022] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.009842] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.031403] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.051045] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.071524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.100330] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.120879] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.151160] Stack:
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.160291]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.180989] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.210148] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.231822] Call Trace:
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.241108]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.261276]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.290047]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.311514]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.340787]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.361900]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.391055]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.411755]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.440991]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.462016]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.490299]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.510665]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.531246]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.560108]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.580368]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.600888]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.620327]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.640680]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.661045]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.680468] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.741693] Call Trace:
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.751105]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.771467]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.800179]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.821662]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.850594]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.871835]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.901161]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.921851]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.961676]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17278.981345]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.001189]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.021535]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.050304]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.070845]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.090647]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.111212]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.130699]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.151162]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:56:47 atlas-tds-mds1 kernel: [17279.171261]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.271079] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.291097] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.561843] CPU 9 
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.562907] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.841184] 
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.842470] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.881534] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.902802] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.922230] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.951143] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.971751] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17361.992464] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.021397] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.042010] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.072045] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.091443] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.111957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.132685] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.161633] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.191640] Stack:
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.193187]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.221455] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.242605] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.272248] Call Trace:
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.281317]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.301682]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.322138]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.352134]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.372848]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.402121]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.423309]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.452504]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.473461]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.502403]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.522357]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.542715]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.571623]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.592189]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.612375]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.632695]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.652201]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.672626]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.701446]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.712698] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.781664] Call Trace:
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.783490]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.811694]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.832074]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.861916]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.882663]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.912171]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.933355]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.962340]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17362.983241]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.012333]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.032538]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.052541]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.081601]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.102050]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.122046]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.142405]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.161805]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.182207]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:58:11 atlas-tds-mds1 kernel: [17363.202576]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.302199] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.322231] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.592715] CPU 9 
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.593893] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.863372] 
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.872485] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.903708] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.933238] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.952447] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.973082] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17445.993749] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.022705] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.043445] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.064168] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.094160] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.113493] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.142589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.163368] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.184118] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.214101] Stack:
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.223351]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.243848] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.272986] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.294485] Call Trace:
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.303760]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.324130]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.352806]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.374318]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.413871]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.433981]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.463217]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.484163]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.513436]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.534178]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.554268]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.582950]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.603463]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.624183]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.644234]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.672930]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.684090]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.713014]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.733350]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.752742] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.813866] Call Trace:
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.823261]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.843602]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.863928]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.893614]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.914330]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.943829]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.973256]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17446.993849]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.014678]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.044052]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.063824]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.084323]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.113114]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.133680]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.153868]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.174241]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.193664]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.214027]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 18:59:35 atlas-tds-mds1 kernel: [17447.234321]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 18:59:36 atlas-tds-mds1 kernel: [17447.711536] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 18:59:36 atlas-tds-mds1 kernel: [17447.744327] LustreError: Skipped 3 previous similar messages
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.333308] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.353551] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.614947] CPU 9 
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.623832] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.894330] 
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.895579] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.934598] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.955863] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17529.974940] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.003842] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.024361] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.045306] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.074286] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.094611] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.124593] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.144052] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.164797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.185339] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.214459] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.244476] Stack:
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.246114]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.274251] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.295241] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.325027] Call Trace:
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.334317]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.354747]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.375218]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.405115]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.425784]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.455165]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.484367]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.505159]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.534470]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.555194]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.575283]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.595377]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.624466]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.645141]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.665156]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.685480]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.705035]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.725575]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.754269]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.765494] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.834680] Call Trace:
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.844114]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.864717]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.885069]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.914931]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.935645]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.965204]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17530.994605]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.015212]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.036222]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.065351]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.085371]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.105670]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.134456]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.154906]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.175060]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.195491]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.215029]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.235549]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:00:59 atlas-tds-mds1 kernel: [17531.264291]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.364584] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.384884] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.655265] CPU 9 
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.656345] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.936231] 
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.945293] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17613.976743] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.006315] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.025543] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.046229] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.075167] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.095894] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.116454] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.145516] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.167040] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.186599] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.215640] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.236220] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.256682] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.286752] Stack:
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.295638]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.316512] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.345858] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.375265] Call Trace:
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.377137]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.405474]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.425874]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.455663]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.476472]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.505866]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.526793]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.555885]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.576835]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.605892]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.625952]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.646317]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.667041]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.695858]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.716027]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.736375]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.755829]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.776308]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.796520]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.815988] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.877107] Call Trace:
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.886387]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.906744]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.935728]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.957123]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17614.986323]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.007368]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.036874]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.057345]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.086875]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.115965]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.135950]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.156129]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.176694]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.197061]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.225609]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.246025]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.265645]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.286053]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:02:23 atlas-tds-mds1 kernel: [17615.306406]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.395863] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.416424] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.686314] CPU 9 
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.687397] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.957240] 
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.958449] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17697.997422] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.027043] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.037965] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.066836] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.087403] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.108136] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.136893] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.157582] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.187275] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.206715] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.227465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.256546] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.277130] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.307373] Stack:
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.308912]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.337223] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.358142] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.387999] Call Trace:
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.397219]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.417438]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.437821]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.467813]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.488340]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.517824]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.547079]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.567771]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.588469]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.617834]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.637638]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.658180]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.686902]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.707324]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.727520]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.747862]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.767190]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.787707]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.807915]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.827656] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.888596] Call Trace:
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.897892]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.918099]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.946882]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17698.968240]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.007719]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.028177]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.057690]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.078498]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.107673]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.128520]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.148184]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.176919]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.197320]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.218020]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.238038]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.266922]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.278052]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.306972]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:03:47 atlas-tds-mds1 kernel: [17699.327338]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.427128] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.447569] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.718052] CPU 9 
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.719060] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.989188] 
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17781.998503] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.037946] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.059286] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.078714] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.099372] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.128635] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.149065] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.178041] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.198557] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.228396] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.247836] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.268643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.289150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.318105] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.339857] Stack:
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.348806]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.377898] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.399046] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.428725] Call Trace:
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.437923]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.458329]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.478563]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.508338]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.528940]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.558404]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.579468]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.608572]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.629307]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.658664]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.678486]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.699035]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.719380]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.748224]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.768306]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.788789]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.808265]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.828694]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.849029]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.868515] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.929422] Call Trace:
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.938835]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.959179]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17782.979615]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.009260]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.030032]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.059492]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.088761]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.109320]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.130274]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.159348]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.179178]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.199338]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.228257]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.248641]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.268833]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.289205]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.308647]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.329066]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:05:11 atlas-tds-mds1 kernel: [17783.349384]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:06:03 atlas-tds-mds1 kernel: [17835.560322] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 19:06:03 atlas-tds-mds1 kernel: [17835.588307] Lustre: Skipped 17 previous similar messages
Mar 31 19:06:04 atlas-tds-mds1 kernel: [17835.599376] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:06:04 atlas-tds-mds1 kernel: [17835.648570] Lustre: Skipped 15 previous similar messages
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17865.458378] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17865.478894] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17865.749352] CPU 9 
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17865.750416] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.020116] 
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.029267] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.060528] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.090063] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.109339] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.129887] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.150606] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.179653] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.199917] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.229140] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.250677] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.270130] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.299182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.319779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.340484] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.380878] Stack:
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.382421]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.409268] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.430027] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.459837] Call Trace:
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.461682]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.489632]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.510032]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.539693]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.560415]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.589874]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.610867]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.639901]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.660853]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.689945]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.709747]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.730156]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.750815]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.779594]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.799596]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.819996]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.839388]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.859830]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.880214]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.899868] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.960800] Call Trace:
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.970221]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17866.990741]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.019508]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.041139]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.070251]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.091295]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.120633]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.149876]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.170680]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.191509]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.219729]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.240149]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.260685]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.289563]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.309568]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.329998]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.349641]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.370188]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:06:35 atlas-tds-mds1 kernel: [17867.390368]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17949.489615] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17949.501933] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17949.780697] CPU 9 
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17949.781707] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.060616] 
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.061844] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.101084] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.122327] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.141465] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.170429] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.191041] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.211592] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.231923] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.261103] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.290871] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.302072] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.331087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.351482] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.380573] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.402525] Stack:
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.411617]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.440722] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.461824] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.491535] Call Trace:
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.500629]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.521145]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.541479]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.571450]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.592102]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.621429]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.642517]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.671650]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.701056]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.721648]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.741803]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.762150]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.791066]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.811462]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.831501]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.852057]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.871499]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.892120]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.920735]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17950.931852] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.001185] Call Trace:
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.010789]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.031199]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.051745]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.081170]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.102140]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.131822]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.152868]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.181828]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.202815]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.231699]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.251721]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.271969]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.292942]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.321746]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.341497]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.361860]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.381324]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.401727]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:07:59 atlas-tds-mds1 kernel: [17951.421846]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18033.520836] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18033.533024] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18033.812038] CPU 9 
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18033.813043] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.091951] 
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.093181] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.132382] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.162115] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.173032] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.202173] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.222559] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.243382] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.272406] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.292852] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.322646] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.342067] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.362588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.391852] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.412440] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.442666] Stack:
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.444341]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.472416] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.493520] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.523215] Call Trace:
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.532482]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.552901]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.573478]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.603149]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.632347]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.653313]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.682720]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.703520]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.732782]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.753730]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.773429]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.802257]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.822778]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.843460]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.863625]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.892038]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.903184]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.932050]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.952484]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18034.963475] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.032678] Call Trace:
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.042130]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.062716]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.083209]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.112956]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.133763]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.163150]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.192519]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.213167]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.242548]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.263369]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.283287]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.303680]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.332708]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.353250]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.373244]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.393479]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.413208]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.433672]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:09:23 atlas-tds-mds1 kernel: [18035.462274]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18117.552044] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18117.564298] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18117.834051] CPU 9 
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18117.842979] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.113671] 
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.114957] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.153889] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.183455] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.194360] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.223286] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.243906] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.264386] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.293559] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.314094] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.343973] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.363358] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.384110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.413159] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.433769] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.463980] Stack:
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.465523]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.493802] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.514676] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.544746] Call Trace:
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.554218]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.574408]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.594693]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.624597]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.653881]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.675087]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.704309]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.725077]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.754318]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.775131]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.803738]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.823918]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.844649]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.873333]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.893408]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.913548]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.924725]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.964105]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.983579]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18118.994860] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.064301] Call Trace:
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.073637]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.094060]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.114665]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.144247]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.164805]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.194431]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.215289]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.244446]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.265173]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.294291]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.314306]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.334650]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.363564]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.384108]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.404187]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.424468]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.444157]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.464546]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:10:47 atlas-tds-mds1 kernel: [18119.484842]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:12:06 atlas-tds-mds1 kernel: [18198.013662] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 19:12:06 atlas-tds-mds1 kernel: [18198.053288] LustreError: Skipped 4 previous similar messages
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18201.583239] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18201.595738] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18201.874475] CPU 9 
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18201.875480] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.145356] 
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.154534] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.185673] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.215350] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.234631] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.255364] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.275764] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.304847] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.325573] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.354664] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.376168] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.395573] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.424740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.445279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.465975] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.495998] Stack:
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.504872]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.525650] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.554970] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.576626] Call Trace:
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.585844]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.614558]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.634827]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.656384]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.685397]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.706558]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.736181]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.765062]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.785932]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.806448]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.834863]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.855308]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.875998]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.904838]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.924881]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.945269]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.964751]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18202.985191]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.005443]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.024905] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.085966] Call Trace:
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.095416]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.115711]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.136345]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.166252]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.195230]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.216285]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.245767]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.266620]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.295997]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.325231]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.345280]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.365637]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.386148]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.415068]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.435101]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.455755]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.475168]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.495577]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:12:11 atlas-tds-mds1 kernel: [18203.516052]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18285.614420] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18285.626756] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18285.896540] CPU 9 
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18285.905627] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.176137] 
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.177506] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.216359] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.237551] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.256895] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.285934] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.306558] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.327200] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.366701] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.386351] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.416387] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.427391] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.456456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.477369] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.506258] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.536455] Stack:
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.538039]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.565870] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.586805] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.616870] Call Trace:
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.626108]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.646512]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.666888]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.696883]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.717612]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.747104]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.776351]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.797109]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.826385]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.847148]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.867280]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.887428]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.916418]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.936932]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.956983]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.977458]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18286.996976]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.017389]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.046019]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.057142] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.126129] Call Trace:
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.127995]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.156170]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.176568]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.197948]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.227304]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.256526]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.277607]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.306925]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.327651]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.356933]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.376889]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.397117]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.417648]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.446590]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.466447]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.486606]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.506284]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.526734]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:13:35 atlas-tds-mds1 kernel: [18287.546855]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18369.645589] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18369.657935] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18369.928063] CPU 9 
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18369.936948] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.207834] 
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.209131] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.248190] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.277820] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.297056] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.317638] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.338337] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.367256] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.387870] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.408222] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.438305] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.457711] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.478437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.507391] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.528076] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.558126] Stack:
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.567152]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.588103] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.617171] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.638545] Call Trace:
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.647973]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.668348]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.688731]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.718754]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.747833]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.768824]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.798063]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.819040]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.848122]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.868859]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.888966]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.917522]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.938190]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.958752]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18370.978806]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.007555]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.018767]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.047578]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.067975]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.087524] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.148678] Call Trace:
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.158000]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.178556]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.198866]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.228845]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.257909]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.278967]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.308234]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.329120]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.358434]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.379282]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.407679]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.428107]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.448626]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.468937]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.497526]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.517934]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.537540]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.568443]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:14:59 atlas-tds-mds1 kernel: [18371.587910]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18453.676745] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18453.689092] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18453.959255] CPU 9 
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18453.968126] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.238743] 
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.240011] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.279109] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.308780] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.319660] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.348610] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.369353] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.390040] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.418821] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.439531] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.469585] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.489027] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.509745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.538650] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.559555] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.589772] Stack:
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.598744]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.619496] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.648695] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.670183] Call Trace:
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.679486]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.699882]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.720002]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.750074]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.779128]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.800343]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.829890]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.850686]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.879760]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.909049]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.929021]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.949445]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.970128]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18454.998733]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.018959]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.039043]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.058722]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.079062]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.099462]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.118866] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.180034] Call Trace:
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.189518]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.209839]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.230042]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.259746]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.280704]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.310259]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.339351]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.360279]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.389561]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.410457]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.430236]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.459052]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.479601]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.500171]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.520301]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.549119]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.560359]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.588954]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:16:23 atlas-tds-mds1 kernel: [18455.609500]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:17:33 atlas-tds-mds1 kernel: [18525.296733] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 19:17:33 atlas-tds-mds1 kernel: [18525.325471] Lustre: Skipped 16 previous similar messages
Mar 31 19:17:33 atlas-tds-mds1 kernel: [18525.336290] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:17:33 atlas-tds-mds1 kernel: [18525.385835] Lustre: Skipped 16 previous similar messages
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18537.707888] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18537.720263] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18537.990335] CPU 9 
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18537.999195] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.269873] 
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.271098] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.310293] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.331536] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.350655] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.379816] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.400384] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.421253] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.449812] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.470733] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.500320] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.519806] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.540545] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.561131] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.590178] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.620395] Stack:
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.621931]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.649999] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.671198] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.701045] Call Trace:
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.710325]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.730682]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.751148]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.780662]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.801381]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.830908]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.870840]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.891050]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.920345]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.940909]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.961087]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18538.981478]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.010395]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.030932]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.051096]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.071463]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.090929]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.111328]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.131467]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.151231] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.220253] Call Trace:
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.222103]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.250253]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.270850]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.300636]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.321626]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.351065]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.371988]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.401096]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.422084]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.451111]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.470928]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.491444]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.520360]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.540759]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.560758]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.581358]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.600964]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.621167]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:17:47 atlas-tds-mds1 kernel: [18539.641625]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18621.739093] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18621.751569] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.021517] CPU 9 
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.022597] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.301258] 
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.302567] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.341425] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.362686] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.381992] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.402487] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.431704] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.452268] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.481110] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.501867] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.531854] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.551277] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.571862] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.592557] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.621489] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.651527] Stack:
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.653068]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.681352] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.702413] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.732071] Call Trace:
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.741230]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.761608]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.782013]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.811768]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.832582]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.862018]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.882943]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.912218]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.932980]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.962214]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18622.982240]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.002570]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.031483]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.051910]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.072130]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.092249]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.112008]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.132556]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.152804]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.172141] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.241380] Call Trace:
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.243029]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.271396]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.292071]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.321824]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.342772]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.372179]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.393073]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.422190]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.443125]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.472291]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.492460]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.512908]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.541712]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.562056]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.582335]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.602628]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.621906]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.642692]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:19:11 atlas-tds-mds1 kernel: [18623.662972]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18705.770297] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18705.782579] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.052440] CPU 9 
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.053445] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.332703] 
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.333928] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.373153] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.402756] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.413566] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.442628] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.463219] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.484018] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.512798] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.533190] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.563025] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.582617] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.603407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.624030] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.653132] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.683025] Stack:
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.684570]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.712873] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.733904] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.763704] Call Trace:
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.772936]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.793334]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.813869]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.843667]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.864385]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.893775]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.923161]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.943925]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.973200]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18706.994083]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.013936]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.042633]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.063210]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.083759]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.103901]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.124167]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.143768]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.164304]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.192718]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.204018] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.273086] Call Trace:
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.282752]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.303135]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.323543]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.353115]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.373911]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.403336]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.424269]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.453333]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.474431]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.503342]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.523379]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.543748]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.564305]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.593171]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.613259]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.633600]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.653215]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.673696]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:20:35 atlas-tds-mds1 kernel: [18707.693975]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18789.801487] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18789.813707] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.083770] CPU 9 
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.084925] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.354968] 
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.363833] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.395126] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.424834] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.444062] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.464790] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.493736] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.514380] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.534952] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.563748] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.585247] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.604860] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.633809] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.654345] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.674941] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.705281] Stack:
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.714358]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.735295] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.764403] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.794398] Call Trace:
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.796261]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.824073]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.844275]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.874277]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.895140]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.924506]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.945705]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.974672]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18790.995873]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.024920]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.044564]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.065083]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.093985]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.114523]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.134729]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.155197]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.174719]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.195035]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.215424]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.235092] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.304232] Call Trace:
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.306165]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.334244]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.354694]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.384431]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.405060]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.434449]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.455408]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.484643]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.515936]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.535233]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.555336]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.575842]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.604751]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.625073]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.645315]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.665607]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.685282]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.705472]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:21:59 atlas-tds-mds1 kernel: [18791.734177]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18873.832664] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18873.845003] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.115475] CPU 9 
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.124515] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.395395] 
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.396621] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.435627] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.456835] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.476162] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.505033] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.525673] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.546424] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.575301] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.595741] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.625671] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.645105] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.665841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.686691] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.715531] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.745915] Stack:
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.747456]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.775521] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.796430] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.826134] Call Trace:
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.835619]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.855810]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.876424]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.906040]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.926929]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.956169]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18874.985631]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.006045]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.027048]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.056057]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.076151]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.096573]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.125486]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.145891]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.166080]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.186718]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.206225]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.226478]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.255401]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.266701] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.335552] Call Trace:
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.337378]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.365604]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.386186]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.416184]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.436720]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.466181]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.487269]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.516220]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.537291]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.566457]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.586239]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.606653]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.635535]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.656296]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.676232]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.696602]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.716531]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.736634]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:23:23 atlas-tds-mds1 kernel: [18875.756977]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:24:36 atlas-tds-mds1 kernel: [18948.286078] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 19:24:36 atlas-tds-mds1 kernel: [18948.323180] LustreError: Skipped 4 previous similar messages
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18957.863829] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18957.876153] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.146622] CPU 9 
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.147764] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.426785] 
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.428013] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.467074] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.496746] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.507607] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.536553] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.557147] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.577799] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.606966] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.627496] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.657370] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.676787] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.697584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.726601] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.747246] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.777423] Stack:
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.786464]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.807404] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.836542] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.857995] Call Trace:
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.867231]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.887539]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.918293]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.937893]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.967169]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18958.988371]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.017516]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.038135]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.067417]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.088156]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.116621]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.137066]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.157739]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.178202]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.206660]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.227174]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.246663]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.267068]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.287569]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.307052] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.367990] Call Trace:
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.377424]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.398003]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.426766]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.448144]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.477279]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.498404]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.527704]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.548624]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.577719]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.598620]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.626767]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.646996]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.667737]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.688204]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.708087]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.736983]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.748441]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.776890]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:24:47 atlas-tds-mds1 kernel: [18959.797409]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:25:06 atlas-tds-mds1 kernel: [18978.345475] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 12@gni2  ns: mdt-atlastds-MDT0000_UUID lock: ffff88075ec42900/0xcfedff8c85b22b1c lrc: 3/0,0 mode: PR/PR res: [0x200003ab1:0x1:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x200000000020 nid: 12@gni2 remote: 0xc2e9e18a619bff0c expref: 667 pid: 12440 timeout: 4313635417 lvb_type: 0 used 0
Mar 31 19:25:06 atlas-tds-mds1 kernel: [18978.466105] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2 previous similar messages
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19041.894981] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19041.907636] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.177742] CPU 9 
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.178908] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.449322] 
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.458425] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.498052] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.519472] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.538770] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.567885] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.588641] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.609271] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.637928] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.658681] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.688292] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.707960] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.728721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.749371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.778412] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.808377] Stack:
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.810049]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.838197] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.859224] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.889120] Call Trace:
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.898419]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.918789]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.939260]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.969168]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19042.989835]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.019212]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.048647]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.069445]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.098491]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.119270]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.139418]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.168017]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.188400]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.209048]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.229203]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.249457]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.268923]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.289407]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.318172]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.329221] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.398168] Call Trace:
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.399999]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.428204]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.448626]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.478412]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.499382]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.528856]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.550014]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.579158]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.599935]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.628898]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.648905]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.669239]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.698125]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.718536]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.738722]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.758969]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.778373]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.798780]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:26:11 atlas-tds-mds1 kernel: [19043.819183]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:27:33 atlas-tds-mds1 kernel: [19125.542940] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 19:27:33 atlas-tds-mds1 kernel: [19125.569174] Lustre: Skipped 15 previous similar messages
Mar 31 19:27:33 atlas-tds-mds1 kernel: [19125.580088] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:27:33 atlas-tds-mds1 kernel: [19125.629239] Lustre: Skipped 15 previous similar messages
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19125.926121] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19125.938775] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.208918] CPU 9 
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.210021] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.480098] 
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.489062] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.520269] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.549813] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.569172] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.589783] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.610291] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.639231] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.659835] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.680315] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.710376] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.729809] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.750389] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.779450] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.800352] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.830511] Stack:
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.839535]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.860333] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.889879] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.911848] Call Trace:
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.920277]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.940815]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.969405]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19126.990901]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.020023]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.041054]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.070688]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.099796]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.120856]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.149878]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.170040]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.190229]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.219300]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.239831]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.259881]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.280384]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.299883]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.320257]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.340595]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.360253] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.429332] Call Trace:
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.431035]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.459421]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.480004]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.509779]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.530500]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.560195]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.581215]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.610182]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.631069]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.660202]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.680200]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.700425]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.729446]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.749832]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.769885]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.790223]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.809692]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.818874] Lustre: lock timed out (enqueued at 1396308155, 300s ago)
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.818911] LustreError: dumping log to /tmp/lustre-log.1396308455.12310
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.870719]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:27:35 atlas-tds-mds1 kernel: [19127.891157]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19209.957249] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19209.970123] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.240627] CPU 9 
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.241629] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.511653] 
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.520585] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.551906] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.581703] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.600950] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.621462] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.650351] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.670984] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.691768] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.720617] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.741904] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.761510] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.790396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.811182] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.831781] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.861943] Stack:
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.871217]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.891883] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.921068] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.950440] Call Trace:
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.952215]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19210.980539]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.001105]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.030914]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.051665]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.081328]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.102298]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.131406]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.152235]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.181389]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.201356]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.221730]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.261117]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.281045]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.301185]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.321559]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.341056]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.361280]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.381900]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.401510] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.462629] Call Trace:
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.471851]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.492148]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.521128]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.542525]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.571786]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.601118]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.622324]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.651390]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.672273]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.701565]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.721489]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.741897]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.770786]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.791398]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.811400]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.831746]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.851429]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.871588]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:28:59 atlas-tds-mds1 kernel: [19211.892104]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19293.988366] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.001427] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.271963] CPU 9 
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.273053] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.543051] 
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.552194] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.591624] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.613112] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.632231] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.652975] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.681682] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.702283] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.723024] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.751914] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.773522] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.793286] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.822211] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.842911] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.871843] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.902178] Stack:
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.903793]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.931931] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.952907] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.982728] Call Trace:
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19294.992027]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.012281]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.032754]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.062626]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.083333]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.112820]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.142255]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.163018]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.183702]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.213066]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.233176]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.253620]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.282488]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.303127]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.323254]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.343483]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.362935]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.383239]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.411979]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.423070] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.492356] Call Trace:
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.502007]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.522258]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.542943]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.572618]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.593359]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.622637]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.643690]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.672650]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.693571]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.722623]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.742637]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.762894]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.783552]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.812324]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.832298]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.852706]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.872225]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.892593]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:30:23 atlas-tds-mds1 kernel: [19295.912970]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.019471] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.032842] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.302900] CPU 9 
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.304081] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.573873] 
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.582945] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.614346] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.643802] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.654700] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.683573] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.714617] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.733583] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.754304] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.783211] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.804836] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.824315] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.853271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.873999] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.894716] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.924585] Stack:
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.933680]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.954518] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19378.983824] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.013131] Call Trace:
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.014978]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.043249]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.063662]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.085266]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.114335]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.143604]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.164844]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.193810]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.214701]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.243791]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.263778]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.284000]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.304527]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.333232]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.353332]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.373705]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.393300]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.413727]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.434074]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.453689] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.514793] Call Trace:
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.524254]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.544647]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.573341]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.595058]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.623959]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.645252]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.674576]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.695245]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.724379]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.745276]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.773448]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.793819]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.814403]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.834849]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.855034]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.883808]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.894979]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.923697]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:31:47 atlas-tds-mds1 kernel: [19379.944220]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:32:38 atlas-tds-mds1 kernel: [19430.354891] Lustre: lock timed out (enqueued at 1396308458, 300s ago)
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.050565] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.064105] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.334393] CPU 9 
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.335462] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.605569] 
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.614690] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.645889] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.675457] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.694697] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.715347] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.744381] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.764753] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.785518] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.814390] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.836318] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.855644] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.884628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.905363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.925983] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.956011] Stack:
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.965090]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19462.985902] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.015055] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.044462] Call Trace:
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.046221]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.074711]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.095099]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.124898]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.145662]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.175114]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.196044]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.225320]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.246219]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.275330]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.295156]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.315675]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.344673]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.365020]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.385199]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.405346]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.425066]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.445406]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.465820]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.485424] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.546693] Call Trace:
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.555764]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.576166]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.604893]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.626530]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.655698]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.676418]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.705671]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.726411]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.755933]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.776676]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.804977]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.825405]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.845844]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.866190]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.896556]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.914990]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.926259]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.954868]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:33:11 atlas-tds-mds1 kernel: [19463.975445]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:33:24 atlas-tds-mds1 kernel: [19476.463956] Lustre: 12132:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply
Mar 31 19:33:24 atlas-tds-mds1 kernel: [19476.463958]   req@ffff880727a7f000 x1463845650165632/t0(0) o101->98846250-6b04-9191-3660-3a16c1e56d27@13@gni2:0/0 lens 656/3512 e 1 to 0 dl 1396308809 ref 2 fl Interpret:/0/0 rc 0/0
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.081648] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.095614] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.365953] CPU 9 
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.367095] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.636714] 
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.645841] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.677085] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.706494] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.725834] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.746603] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.767115] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.796082] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.816632] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.837353] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.867049] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.886485] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.907347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.936165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.956900] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.987064] Stack:
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19546.996154]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.017048] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.046316] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.067774] Call Trace:
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.076951]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.105833]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.126280]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.147629]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.176830]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.197756]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.227244]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.256276]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.277061]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.306346]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.326308]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.346471]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.367055]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.395934]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.416137]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.436699]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.456191]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.476776]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.497050]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.516719] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.577744] Call Trace:
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.587118]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.607449]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.636263]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.657766]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.686819]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.707774]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.737178]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.757855]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.787344]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.808107]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.836300]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.856524]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.877224]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.906074]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.926136]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.946596]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.957577]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19547.986581]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:34:35 atlas-tds-mds1 kernel: [19548.006898]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:34:36 atlas-tds-mds1 kernel: [19548.508654] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 19:34:36 atlas-tds-mds1 kernel: [19548.546769] LustreError: Skipped 3 previous similar messages
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.112721] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.126910] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.397488] CPU 9 
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.398493] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.668533] 
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.677566] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.717170] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.738628] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.757947] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.778428] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.807426] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.828113] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.857224] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.877971] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.907827] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.927292] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.947847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.968344] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19630.997509] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.027469] Stack:
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.029010]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.057152] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.078246] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.107872] Call Trace:
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.117155]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.137584]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.158117]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.187922]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.219169]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.238526]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.267808]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.288714]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.318013]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.338669]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.358806]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.387446]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.408034]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.428369]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.448586]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.477248]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.488548]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.517301]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.537704]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.548953] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.618035] Call Trace:
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.627540]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.647985]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.668303]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.698119]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.718929]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.748353]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.777728]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.798421]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.819190]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.848362]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.868359]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.888547]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.917645]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.938013]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.958247]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.978623]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19631.998290]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19632.018560]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:35:59 atlas-tds-mds1 kernel: [19632.039051]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.143783] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.158266] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.428424] CPU 9 
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.429533] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.699441] 
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.708475] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.739775] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.769275] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.788678] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.809269] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.838336] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.858748] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.879270] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.899809] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.929849] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.949300] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.969803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19714.999037] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.019755] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.049785] Stack:
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.058830]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.079564] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.108804] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.130429] Call Trace:
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.139582]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.159890]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.188671]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.210349]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.239474]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.260344]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.289682]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.310169]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.339528]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.360202]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.388723]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.409140]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.429727]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.458585]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.478754]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.499165]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.518636]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.538853]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.559348]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.578765] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.639857] Call Trace:
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.649232]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.669430]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.690055]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.720034]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.749119]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.770052]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.799651]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.820352]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.849687]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.870362]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.890364]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.918921]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.939703]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.960034]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19715.980190]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19716.008781]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19716.020002]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19716.040367]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:37:23 atlas-tds-mds1 kernel: [19716.069027]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:37:33 atlas-tds-mds1 kernel: [19725.769839] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 19:37:33 atlas-tds-mds1 kernel: [19725.793186] Lustre: Skipped 15 previous similar messages
Mar 31 19:37:33 atlas-tds-mds1 kernel: [19725.812392] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:37:33 atlas-tds-mds1 kernel: [19725.853376] Lustre: Skipped 15 previous similar messages
Mar 31 19:38:27 atlas-tds-mds1 kernel: [19779.576022] Lustre: 12277:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply
Mar 31 19:38:27 atlas-tds-mds1 kernel: [19779.576023]   req@ffff88075bb89800 x1463845650239132/t0(0) o101->98846250-6b04-9191-3660-3a16c1e56d27@13@gni2:0/0 lens 576/3384 e 1 to 0 dl 1396309112 ref 2 fl Interpret:/0/0 rc 0/0
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.174834] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.189598] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.461345] CPU 9 
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.470024] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.740744] 
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.749819] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.780946] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.810629] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.829824] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.850424] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.871165] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.900276] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.920810] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.949908] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.971364] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19798.990819] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.019878] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.040335] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.060865] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.090915] Stack:
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.100144]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.120915] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.150152] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.171820] Call Trace:
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.180949]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.201135]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.230001]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.251477]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.280789]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.310237]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.331341]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.360236]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.381224]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.410420]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.430520]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.450747]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.471209]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.500104]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.520098]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.540514]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.559911]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.580353]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.600885]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.620288] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.681108] Call Trace:
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.690545]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.710826]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.731210]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.761022]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.781711]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.811230]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.840418]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.861379]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.890640]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.911509]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.931486]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.960099]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19799.980687]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.001208]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.021232]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.041716]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.061246]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.081689]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:38:47 atlas-tds-mds1 kernel: [19800.110542]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.205875] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.220985] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.491097] CPU 9 
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.492373] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.762237] 
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.771304] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.802364] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.832066] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.851130] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.871961] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.892439] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.921380] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.942142] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.971063] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19882.992507] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.012274] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.041058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.061779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.082482] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.112358] Stack:
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.121252]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.142214] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.171522] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.193232] Call Trace:
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.202403]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.222850]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.251560]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.272952]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.301933]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.322949]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.352630]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.381582]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.402323]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.431757]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.451552]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.471970]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.492644]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.521452]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.541691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.561993]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.581456]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.602033]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.622497]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.642200] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.711324] Call Trace:
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.713193]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.741385]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.761715]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.791695]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.812499]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.841722]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.863043]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.891911]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.912699]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.941925]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.961792]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19883.982160]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.002575]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.031395]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.051482]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.072001]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.091422]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.111674]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:40:11 atlas-tds-mds1 kernel: [19884.132009]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.236907] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.252042] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.513961] CPU 9 
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.522691] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.793561] 
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.802598] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.833717] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.863302] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.882532] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.903322] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.923819] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.952909] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19966.973436] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.002328] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.024034] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.043554] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.072569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.093249] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.113873] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.144181] Stack:
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.153179]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.173974] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.203433] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.232418] Call Trace:
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.234420]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.262660]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.282870]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.304293]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.333445]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.363032]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.384021]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.413086]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.434206]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.463088]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.482911]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.503479]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.523999]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.552763]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.573023]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.593524]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.612948]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.633370]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.653688]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.673327] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.734542] Call Trace:
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.743746]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.763918]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.792674]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.814122]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.843262]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.864622]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.893796]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.914676]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.943891]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.964526]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19967.992926]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.013638]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.034067]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.062904]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.083153]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.103607]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.123195]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.143409]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:41:35 atlas-tds-mds1 kernel: [19968.163895]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.150816] LNet: Service thread pid 12310 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.197169] Pid: 12310, comm: mdt00_244
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.206869] 
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.206870] Call Trace:
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.225801]  [<ffffffffa03de76e>] ? cfs_waitq_del+0xe/0x10 [libcfs]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.245783]  [<ffffffffa03de6fe>] cfs_waitq_wait+0xe/0x10 [libcfs]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.265934]  [<ffffffffa06e509a>] ldlm_completion_ast+0x57a/0x960 [ptlrpc]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.286332]  [<ffffffffa06e07a0>] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.315811]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.335634]  [<ffffffffa06e4748>] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.356147]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.376592]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 19:42:35 atlas-tds-mds1 kernel: [20028.396828]  [<ffffffffa0dbacbe>] mdt_object_lock0+0x28e/0xaf0 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.425465]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.445669]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.466203]  [<ffffffffa0dbb5e4>] mdt_object_lock+0x14/0x20 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.486330]  [<ffffffffa0dbb651>] mdt_object_find_lock+0x61/0x170 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.506838]  [<ffffffffa0de5bac>] mdt_reint_open+0x8cc/0x2200 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.526753]  [<ffffffffa03fa83e>] ? upcall_cache_get_entry+0x28e/0x860 [libcfs]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.556252]  [<ffffffffa070df6c>] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.576699]  [<ffffffffa05a4200>] ? lu_ucred+0x20/0x30 [obdclass]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.596427]  [<ffffffffa0db0315>] ? mdt_ucred+0x15/0x20 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.616575]  [<ffffffffa0dcc62c>] ? mdt_root_squash+0x2c/0x410 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.636589]  [<ffffffffa0dd0d21>] mdt_reint_rec+0x41/0xe0 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.656451]  [<ffffffffa0db5b73>] mdt_reint_internal+0x4c3/0x780 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.685344]  [<ffffffffa0db60fd>] mdt_intent_reint+0x1ed/0x520 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.705591]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.726007]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.746514]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.766663]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.786891]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.815407]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.836727]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.856522]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.886055]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.905664]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.936816]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.955663]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.967050]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20028.995475]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.015448]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.026399]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.046745]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.066760]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.086236] 
Mar 31 19:42:36 atlas-tds-mds1 kernel: [20029.095470] LustreError: dumping log to /tmp/lustre-log.1396309356.12310
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.267929] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.283399] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.553861] CPU 9 
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.555013] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.824876] 
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.833956] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.865167] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.894978] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.914249] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.934973] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.963689] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20050.984447] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.005234] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.034032] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.055463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.074933] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.103894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.124431] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.145097] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.175059] Stack:
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.184305]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.204994] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.234292] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.255717] Call Trace:
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.264846]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.285286]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.314132]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.335464]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.364537]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.385673]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.414961]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.435716]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.464860]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.485702]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.513845]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.534106]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.554816]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.575178]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.595180]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.615544]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.635130]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.655514]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.684100]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.695183] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.764495] Call Trace:
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.774105]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.794540]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.814767]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.844752]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.865438]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.894960]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.915890]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.945064]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.966020]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20051.994830]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.014784]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.035010]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.055655]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.084456]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.104699]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.125173]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.144684]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.165131]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:42:59 atlas-tds-mds1 kernel: [20052.185575]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.298941] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.314930] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.585360] CPU 9 
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.594425] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.856625] 
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.865666] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.905110] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.926711] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.945729] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.966375] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20134.995316] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.015901] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.036439] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.065584] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.095540] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.115019] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.135744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.156480] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.185194] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.215443] Stack:
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.216984]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.245193] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.265991] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.295861] Call Trace:
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.297578]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.325372]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.345727]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.375512]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.396426]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.436209]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.456453]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.485523]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.506321]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.535556]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.555614]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.575614]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.596293]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.616848]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.636900]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.665635]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.676770]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.705416]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.725831]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.745225] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.806196] Call Trace:
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.815650]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.836083]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.856559]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.886455]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.907275]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.936677]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.966310]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20135.986989]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.016198]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.037030]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.065514]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.085790]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.106319]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.135390]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.155567]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.175975]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.195440]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.215801]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:44:23 atlas-tds-mds1 kernel: [20136.236190]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.329944] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.346206] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.616408] CPU 9 
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.617501] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.887219] 
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.888417] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.927506] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.957229] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.976475] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20218.996908] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.017591] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.046524] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.067281] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.087954] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.117849] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.137125] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.157739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.186770] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.207262] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.237146] Stack:
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.246375]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.267128] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.296405] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.317857] Call Trace:
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.326988]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.347384]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.367842]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.398027]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.427287]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.448497]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.477946]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.506839]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.527796]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.556844]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.576684]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.597135]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.617809]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.646587]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.666591]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.687111]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.706619]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.726979]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.747346]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.766748] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.827898] Call Trace:
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.837178]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.857827]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.878101]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.907789]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.928598]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.957988]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20219.987591]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.008172]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.037653]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.058398]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.086714]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.107139]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.127533]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.148017]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.167990]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.188323]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.207850]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.228202]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:45:47 atlas-tds-mds1 kernel: [20220.256965]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:47:06 atlas-tds-mds1 kernel: [20298.767898] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 19:47:06 atlas-tds-mds1 kernel: [20298.806559] LustreError: Skipped 4 previous similar messages
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.360938] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.377497] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.647796] CPU 9 
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.648946] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.918914] 
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.927944] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.959003] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20302.988743] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.008027] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.028736] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.057803] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.078373] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.098893] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.128001] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.149436] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.168965] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.197917] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.218477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.239088] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.269148] Stack:
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.278236]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.299030] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.328284] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.357696] Call Trace:
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.359614]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.387967]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.408102]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.429440]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.458873]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.488366]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.509457]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.538376]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.559371]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.588408]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.608181]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.628611]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.648965]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.677860]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.697831]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.718221]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.729341]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.758042]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.778614]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.798105] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.859068] Call Trace:
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.868547]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.888871]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.909087]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.938884]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.959789]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20303.989159]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.018503]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.039294]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.068584]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.089457]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.109448]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.138038]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.158772]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.179335]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.199482]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.228584]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.239582]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.268222]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:47:11 atlas-tds-mds1 kernel: [20304.288466]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.687444] LNet: Service thread pid 12108 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.739568] Pid: 12108, comm: mdt00_042
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.749209] 
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.749210] Call Trace:
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.768099]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.788739]  [<ffffffffa03de6fe>] cfs_waitq_wait+0xe/0x10 [libcfs]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.808781]  [<ffffffffa06e509a>] ldlm_completion_ast+0x57a/0x960 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.829221]  [<ffffffffa06e07a0>] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.858944]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.878538]  [<ffffffffa06e4748>] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.899275]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.927999]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.948385]  [<ffffffffa0dbad6d>] mdt_object_lock0+0x33d/0xaf0 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.968578]  [<ffffffffa0db0d90>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20330.988766]  [<ffffffffa06e4b20>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.009192]  [<ffffffffa0dbb5e4>] mdt_object_lock+0x14/0x20 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.029121]  [<ffffffffa0dc57d9>] mdt_getattr_name_lock+0xe29/0x1990 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.058121]  [<ffffffffa070d2d5>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.078312]  [<ffffffffa070f564>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.098686]  [<ffffffffa0dc65dd>] mdt_intent_getattr+0x29d/0x490 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.119003]  [<ffffffffa0db3fae>] mdt_intent_policy+0x39e/0x720 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.139585]  [<ffffffffa06c5841>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.168109]  [<ffffffffa06ec2cf>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.188784]  [<ffffffffa0db4436>] mdt_enqueue+0x46/0xe0 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.208836]  [<ffffffffa0db8bc7>] mdt_handle_common+0x647/0x16d0 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.229097]  [<ffffffffa070ed4c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.249936]  [<ffffffffa0df4ae5>] mds_regular_handle+0x15/0x20 [mdt]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.278134]  [<ffffffffa071e568>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.299590]  [<ffffffffa03de5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.319099]  [<ffffffffa03efd9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.348129]  [<ffffffffa07158c9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.368533]  [<ffffffff81055cc3>] ? __wake_up+0x53/0x70
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.379722]  [<ffffffffa071f8fe>] ptlrpc_main+0xace/0x1700 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.408176]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.428354]  [<ffffffff8100c0ca>] child_rip+0xa/0x20
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.439262]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.459339]  [<ffffffffa071ee30>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.479283]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.498933] 
Mar 31 19:47:38 atlas-tds-mds1 kernel: [20331.508177] LustreError: dumping log to /tmp/lustre-log.1396309658.12108
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.384338] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.388127] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.388130] Lustre: Skipped 20 previous similar messages
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391923] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391924] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391957] CPU 9 
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391957] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391983] 
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391984] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391987] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391990] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391992] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391993] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391994] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391995] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391997] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.391998] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392000] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392001] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392005] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392005] Stack:
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392006]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392008] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392011] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392013] Call Trace:
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392045]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392066]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392087]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392099]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392120]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392141]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392150]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392170]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392192]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392204]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392213]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392239]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392263]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392266]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392288]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392291]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392313]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392334]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392336]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392337] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392349] Call Trace:
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392369]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392390]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392410]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392420]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392440]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392461]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392470]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392491]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392513]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392522]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392531]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392551]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392573]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392576]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392597]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392599]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392621]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392643]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20386.392645]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:48:35 atlas-tds-mds1 kernel: [20388.420840] Lustre: Skipped 22 previous similar messages
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.422900] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.439977] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.710068] CPU 9 
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.711189] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.981307] 
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20470.990365] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.021792] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.051516] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.070809] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.091495] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.120249] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.140930] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.161562] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.190617] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.212163] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.231396] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.260464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.281119] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.301692] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.331844] Stack:
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.340673]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.361622] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.390703] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.412187] Call Trace:
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.421469]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.441810]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.470604]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.491869]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.521034]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.542245]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.571379]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.592325]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.621593]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.642388]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.670481]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.690696]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.711308]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.731593]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.751764]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.780683]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.791778]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.820549]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.841047]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.851945] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.921502] Call Trace:
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.930961]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.951325]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20471.971836]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.001554]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.022394]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.051521]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.080997]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.101756]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.131210]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.151802]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.171883]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.192073]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.221103]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.241561]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.261650]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.282012]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.301426]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.321859]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:49:59 atlas-tds-mds1 kernel: [20472.342210]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20554.453868] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20554.471296] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20554.741739] CPU 9 
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20554.742891] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.021822] 
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.023093] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.062077] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.083256] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.102654] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.123108] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.152119] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.172758] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.201780] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.222336] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.252119] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.271754] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.292171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.313153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.342006] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.372007] Stack:
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.373461]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.401830] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.422636] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.452612] Call Trace:
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.461687]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.482336]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.502683]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.532468]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.553413]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.582896]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.612247]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.633279]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.662284]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.683242]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.703176]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.731799]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.752363]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.773029]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.793029]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.813341]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.832790]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.853110]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.881848]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.893009] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.962222] Call Trace:
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.971829]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20555.992461]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.013123]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.042504]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.063225]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.092655]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.113999]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.142956]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.163628]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.192925]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.212862]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.233208]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.262352]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.282863]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.302913]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.323235]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.343082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.363421]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:51:23 atlas-tds-mds1 kernel: [20556.391984]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20638.484877] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20638.502758] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20638.772714] CPU 9 
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20638.773813] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.043712] 
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.052764] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.084170] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.113803] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.133224] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.153755] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.174308] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.203248] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.223815] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.244426] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.274403] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.293847] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.322901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.343507] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.363869] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.394268] Stack:
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.403149]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.424061] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.453375] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.474869] Call Trace:
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.483970]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.504184]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.524518]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.554454]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.583412]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.604525]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.634184]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.654832]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.684209]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.704870]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.733282]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.753666]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.774196]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.805022]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.814807]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.843391]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.854559]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.883320]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.903687]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.923123] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.984123] Call Trace:
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20639.993587]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.013949]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.034311]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.064327]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.093585]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.114548]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.144224]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.164626]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.194049]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.214945]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.234871]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.263650]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.284223]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.304691]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.333317]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.353686]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.373302]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.393558]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:52:47 atlas-tds-mds1 kernel: [20640.413688]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20722.515930] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20722.533873] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20722.804266] CPU 9 
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20722.805493] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.075024] 
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.084069] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.115218] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.144848] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.155606] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.184497] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.205308] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.234161] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.254862] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.275408] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.305302] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.324909] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.345438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.374416] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.395135] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.425185] Stack:
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.434236]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.454993] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.484216] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.505836] Call Trace:
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.515109]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.535471]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.555798]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.585790]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.614892]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.635979]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.665291]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.685987]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.715497]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.736141]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.764380]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.784749]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.805341]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.826012]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.845857]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.874666]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.885881]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.914619]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.935093]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20723.954578] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.015408] Call Trace:
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.024851]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.045204]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.065780]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.095430]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.116188]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.145622]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.175112]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.195831]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.225144]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.245879]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.265892]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.286131]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.315156]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.335520]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.355713]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.376240]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.395688]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.415934]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:54:11 atlas-tds-mds1 kernel: [20724.444806]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20806.546973] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20806.565244] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20806.835475] CPU 9 
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20806.836630] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.106508] 
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.115598] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.146708] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.176476] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.195585] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.216199] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.236875] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.276216] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.295876] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.316592] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.346615] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.366091] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.386628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.415698] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.436545] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.466483] Stack:
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.475538]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.496268] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.525546] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.547050] Call Trace:
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.556184]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.576838]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.596950]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.626754]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.647628]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.677119]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.706365]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.727137]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.756269]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.776993]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.796767]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.825693]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.846064]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.866633]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.886966]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.915708]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.927009]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.955753]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.976180]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20807.987295] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.056705] Call Trace:
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.066147]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.086543]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.106733]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.136581]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.157337]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.186827]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.216215]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.237102]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.266475]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.287132]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.307083]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.335883]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.356265]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.376958]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.397012]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.425912]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.437258]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.457595]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:55:35 atlas-tds-mds1 kernel: [20808.486129]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20890.578006] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20890.596556] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20890.867006] CPU 9 
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20890.868164] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.138132] 
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.147280] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.186712] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.208043] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.227467] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.248110] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.277192] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.297751] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.318197] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.347178] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.368592] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.388201] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.417020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.437755] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.466843] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.488479] Stack:
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.497590]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.518419] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.547791] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.577436] Call Trace:
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.579155]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.607314]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.627315]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.648878]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.678077]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.707305]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.728402]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.757332]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.778268]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.807330]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.827363]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.847552]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.868327]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.897031]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.917011]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.937299]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.948558]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.977092]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20891.997607]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.017026] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.078014] Call Trace:
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.087454]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.108096]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.128408]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.158040]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.178961]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.208263]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.237709]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.258595]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.287989]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.308705]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.328547]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.357368]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.377772]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.398507]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.418655]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.447212]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.468831]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.487416]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:56:59 atlas-tds-mds1 kernel: [20892.507762]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20974.609030] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20974.627958] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20974.898319] CPU 9 
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20974.899582] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.169433] 
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.178743] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.218239] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.239655] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.259113] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.279296] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.308515] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.329122] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.349606] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.378634] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.408523] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.419668] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.448499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.469360] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.498411] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.520082] Stack:
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.528911]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.549629] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.579075] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.608752] Call Trace:
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.610614]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.638214]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.658792]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.680003]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.709428]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.738831]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.759882]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.788803]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.809913]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.839014]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.858870]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.879457]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.908333]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.928699]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.948711]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.969238]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20975.988728]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.009147]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.029320]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.048776] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.109653] Call Trace:
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.119163]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.139433]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.159914]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.189525]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.218812]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.239900]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.269212]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.290075]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.319241]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.339925]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.360081]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.388487]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.409260]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.429734]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.449969]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.478930]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.490000]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.518922]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:58:23 atlas-tds-mds1 kernel: [20976.539280]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:58:33 atlas-tds-mds1 kernel: [20986.668962] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 19:58:33 atlas-tds-mds1 kernel: [20986.693066] Lustre: Skipped 19 previous similar messages
Mar 31 19:58:33 atlas-tds-mds1 kernel: [20986.712279] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 19:58:33 atlas-tds-mds1 kernel: [20986.753124] Lustre: Skipped 21 previous similar messages
Mar 31 19:59:36 atlas-tds-mds1 kernel: [21049.044937] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 19:59:36 atlas-tds-mds1 kernel: [21049.077381] LustreError: Skipped 4 previous similar messages
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21058.640044] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21058.659150] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21058.929566] CPU 9 
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21058.930721] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.200424] 
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.209692] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.249322] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.270811] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.289946] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.310810] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.339721] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.360383] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.380848] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.409768] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.431420] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.450731] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.479773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.500531] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.529448] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.551180] Stack:
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.560223]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.580943] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.610221] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.640021] Call Trace:
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.641893]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.669754]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.690119]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.719954]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.751271]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.770877]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.800092]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.820872]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.850107]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.870878]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.890840]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.919632]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.940175]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.961102]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21059.980917]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.009623]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.020740]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.041056]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.069821]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.080894] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.150222] Call Trace:
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.159690]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.180270]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.200610]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.230472]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.251032]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.280543]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.301687]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.330536]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.351645]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.380578]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.400749]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.421102]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.450206]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.470889]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.490902]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.511278]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.530780]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.550995]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 19:59:47 atlas-tds-mds1 kernel: [21060.579860]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21142.671050] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21142.690634] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21142.960510] CPU 9 
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21142.961590] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.231737] 
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.240627] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.271970] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.301594] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.320678] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.341279] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.361932] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.390884] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.411702] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.432211] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.462197] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.481902] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.510761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.531525] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.552054] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.582116] Stack:
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.591338]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.612023] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.641181] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.662600] Call Trace:
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.671920]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.692395]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.721077]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.742518]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.771620]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.792658]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.821984]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.851227]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.872006]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.901245]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.921278]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.941536]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.962055]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21143.990922]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.011125]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.031658]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.051126]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.071514]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.091878]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.111332] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.172386] Call Trace:
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.181643]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.202264]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.231086]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.252547]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.281977]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.302850]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.332373]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.361423]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.382460]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.411444]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.431281]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.451908]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.472450]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.501138]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.521300]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.541715]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.561154]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.581700]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:01:11 atlas-tds-mds1 kernel: [21144.602133]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21226.702047] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21226.721729] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21226.992226] CPU 9 
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21226.993236] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.272681] 
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.273960] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.313219] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.342920] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.362346] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.382732] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.403413] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.432415] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.452962] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.473449] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.503154] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.522477] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.543158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.572122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.592865] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.622822] Stack:
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.624468]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.652469] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.673352] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.703051] Call Trace:
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.712102]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.732562]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.752890]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.782720]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.803486]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.832966]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.854120]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.883175]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.904024]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.933152]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.953124]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21227.973619]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.002289]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.022763]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.043015]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.063328]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.082928]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.103215]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.123507]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.143289] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.203938] Call Trace:
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.213397]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.233765]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.262299]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.284056]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.312937]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.334027]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.363451]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.384231]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.413553]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.442763]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.462993]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.483133]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.503695]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.532603]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.552633]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.572993]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.592497]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.612888]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:02:35 atlas-tds-mds1 kernel: [21228.633185]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21310.733035] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21310.753047] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.023475] CPU 9 
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.024632] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.294865] 
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.303996] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.343808] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.364995] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.384226] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.404678] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.433901] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.454551] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.483279] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.503898] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.533844] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.553344] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.574055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.594671] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.623514] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.645373] Stack:
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.654433]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.683337] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.704312] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.734180] Call Trace:
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.743464]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.763930]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.784177]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.814000]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.834736]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.864063]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.885198]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.914275]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.935336]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.964473]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21311.984457]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.004942]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.033736]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.054289]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.074479]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.094841]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.114545]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.134809]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.163574]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.174856] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.243950] Call Trace:
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.253602]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.274058]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.294338]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.324228]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.345003]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.374430]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.405948]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.424608]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.445588]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.474804]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.494771]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.515267]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.544096]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.564666]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.584809]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.605206]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.624673]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.645157]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:03:59 atlas-tds-mds1 kernel: [21312.673904]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21394.764015] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21394.784322] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.054428] CPU 9 
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.055601] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.325620] 
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.334727] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.365870] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.395670] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.414953] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.435512] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.464641] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.485306] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.505806] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.534852] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.556269] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.575933] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.604798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.625579] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.646081] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.676412] Stack:
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.685503]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.706219] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.735623] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.765237] Call Trace:
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.767103]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.794748]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.815209]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.836459]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.865685]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.886718]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.916062]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.945174]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.966085]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21395.995185]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.014962]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.035493]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.055936]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.076457]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.105034]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.125219]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.136341]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.165065]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.185619]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.204843] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.265997] Call Trace:
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.275293]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.296004]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.316456]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.346279]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.375335]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.396588]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.425895]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.446601]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.475750]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.496628]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.516704]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.545244]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.565791]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.586338]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.606443]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.635212]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.646619]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.675212]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:05:23 atlas-tds-mds1 kernel: [21396.695611]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21478.794986] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21478.807055] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.077323] CPU 9 
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.086293] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.356756] 
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.365808] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.397119] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.426442] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.437335] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.466278] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.486911] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.516126] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.536664] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.557317] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.587248] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.606495] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.627183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.656149] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.676898] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.706977] Stack:
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.715975]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.736915] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.765965] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.787681] Call Trace:
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.796810]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.827829]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.846245]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.867673]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.897009]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.917872]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.947599]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.976513]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21479.997534]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.026639]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.046681]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.067080]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.087765]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.116294]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.136527]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.156753]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.176367]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.196810]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.217118]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.236611] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.297620] Call Trace:
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.306922]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.327316]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.347785]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.377541]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.406835]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.428038]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.457211]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.477910]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.507337]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.528178]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.556337]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.576490]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.597081]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.617764]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.637819]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.666487]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.677761]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.706566]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:06:47 atlas-tds-mds1 kernel: [21480.726941]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21562.825949] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21562.838245] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.108156] CPU 9 
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.117055] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.387719] 
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.388942] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.428381] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.457758] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.477182] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.497849] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.518479] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.547505] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.568161] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.588770] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.618720] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.638017] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.658725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.687824] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.708205] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.738421] Stack:
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.747652]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.768429] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.797476] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.819217] Call Trace:
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.828348]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.848784]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.877600]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.899172]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.928162]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.949193]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21563.978768]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.007831]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.028724]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.057811]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.077829]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.098023]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.118702]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.138931]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.167454]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.187977]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.207497]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.228055]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.248375]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.268082] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.329012] Call Trace:
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.338290]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.358822]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.379139]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.409052]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.438119]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.459162]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.488591]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.509377]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.538385]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.559196]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.579087]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.607814]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.628442]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.649037]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.669109]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.697813]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.709092]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.737878]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:08:11 atlas-tds-mds1 kernel: [21564.758255]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:09:09 atlas-tds-mds1 kernel: [21622.778701] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 20:09:09 atlas-tds-mds1 kernel: [21622.810008] Lustre: Skipped 20 previous similar messages
Mar 31 20:09:09 atlas-tds-mds1 kernel: [21622.829270] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) refused reconnection, still busy with 2 active RPCs
Mar 31 20:09:09 atlas-tds-mds1 kernel: [21622.870303] Lustre: Skipped 20 previous similar messages
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21646.856922] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21646.869193] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.139985] CPU 9 
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.148885] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.419472] 
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.428633] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.459896] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.489423] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.508874] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.529615] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.558564] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.579219] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.599972] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.628921] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.650526] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.669951] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.698821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.719537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.740224] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.770207] Stack:
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.779294]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.800005] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.829190] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.858609] Call Trace:
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.860369]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.888895]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.909096]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.939050]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.959989]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21647.989250]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.010318]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.039514]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.060606]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.089642]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.109648]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.129882]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.158928]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.179294]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.199694]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.219857]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.239661]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.260014]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.280338]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.299890] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.368995] Call Trace:
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.370905]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.399024]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.419579]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.449410]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.470375]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.499626]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.529245]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.550012]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.571008]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.599660]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.619712]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.640191]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.669135]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.689681]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.709919]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.730127]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.749806]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.770142]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:09:35 atlas-tds-mds1 kernel: [21648.790518]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:09:36 atlas-tds-mds1 kernel: [21649.283708] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 20:09:36 atlas-tds-mds1 kernel: [21649.319560] LustreError: Skipped 3 previous similar messages
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21730.887927] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21730.899934] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.169993] CPU 9 
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.171001] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.441119] 
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.450383] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.489761] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.511266] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.530358] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.551120] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.579957] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.600574] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.621060] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.650032] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.671324] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.690958] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.720052] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.740436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.761112] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.791298] Stack:
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.800112]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.821037] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.850073] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.871808] Call Trace:
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.880852]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.901196]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.929949]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.951442]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21731.980529]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.001928]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.031262]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.060397]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.081482]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.110533]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.130365]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.150814]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.171305]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.200046]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.220040]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.240429]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.260083]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.280418]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.311340]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.321170] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.390274] Call Trace:
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.392135]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.420528]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.441074]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.470941]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.491761]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.521189]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.550550]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.571305]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.600772]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.621526]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.641540]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.661797]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.690608]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.710982]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.731167]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.751701]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.771024]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.791563]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:10:59 atlas-tds-mds1 kernel: [21732.812033]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21814.918923] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21814.931011] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.192481] CPU 9 
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.201346] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.471847] 
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.473150] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.512184] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.541838] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.561095] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.581857] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.602363] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.631294] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.651946] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.672391] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.702446] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.721890] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.742437] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.771170] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.791917] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.821970] Stack:
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.831143]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.851955] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.881175] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.902931] Call Trace:
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.912026]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.932344]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.952759]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21815.982734]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.011682]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.032948]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.062077]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.082799]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.112113]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.133122]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.161424]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.181535]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.202115]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.222686]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.242724]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.271383]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.282813]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.311567]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.331986]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.351567] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.412578] Call Trace:
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.421850]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.442404]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.462568]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.492585]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.513405]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.542637]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.572272]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.593011]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.622106]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.642821]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.662884]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.691499]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.712109]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.732648]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.752990]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.781520]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.792682]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.821586]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:12:23 atlas-tds-mds1 kernel: [21816.841963]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21898.949912] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21898.962167] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.232695] CPU 9 
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.233850] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.503076] 
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.504296] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.543630] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.573192] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.583940] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.612953] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.633498] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.653970] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.682993] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.703535] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.733399] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.763492] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.783013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.803568] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.832672] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.862875] Stack:
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.864487]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.892470] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.913621] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.943326] Call Trace:
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.952582]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.972986]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21899.993386]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.023433]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.044099]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.073716]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.094421]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.123590]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.144406]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.173587]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.193693]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.213970]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.242815]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.263390]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.283417]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.303897]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.323317]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.343663]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.364014]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.383422] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.444600] Call Trace:
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.453806]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.474126]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.502903]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.524467]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.553481]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.574548]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.603895]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.624278]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.653605]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.674413]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.694298]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.723031]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.743569]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.764137]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.783977]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.812850]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.824105]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.844283]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:13:47 atlas-tds-mds1 kernel: [21900.873275]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21982.980891] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21982.993624] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.264101] CPU 9 
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.265256] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.535123] 
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.544248] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.583659] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.604817] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.624275] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.645076] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.673943] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.694508] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.715236] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.744194] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.765757] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.785169] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.814206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.834869] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.863826] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.885736] Stack:
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.894732]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.915373] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.944563] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.974355] Call Trace:
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21983.976230]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.004047]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.024414]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.054264]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.075350]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.104707]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.125965]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.154885]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.175872]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.204891]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.224849]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.245582]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.274299]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.294693]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.314898]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.335229]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.354745]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.375105]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.403981]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.415241] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.484368] Call Trace:
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.486091]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.514415]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.534815]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.564622]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.585374]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.614836]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.635908]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.665059]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.686181]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.715143]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.735065]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.755221]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.784128]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.804705]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.824994]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.845225]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.864739]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.885126]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:15:11 atlas-tds-mds1 kernel: [21984.905534]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.011863] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.024757] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.295788] CPU 9 
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.304688] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.575344] 
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.576567] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.615732] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.645376] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.656308] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.685385] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.705967] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.726588] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.755321] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.775963] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.805661] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.825089] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.845624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.866315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.895300] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.917178] Stack:
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.926152]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.955103] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22067.976203] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.006080] Call Trace:
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.015337]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.035708]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.056118]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.085905]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.106886]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.136113]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.165761]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.186322]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.215628]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.236307]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.256194]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.276739]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.305459]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.326013]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.346181]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.366544]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.386017]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.406415]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.426735]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.446512] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.515725] Call Trace:
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.525344]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.545736]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.566126]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.595880]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.616857]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.646335]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.667267]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.696388]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.725845]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.746573]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.766756]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.786927]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.815889]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.836382]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.856230]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.876723]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.896233]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.917065]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:16:35 atlas-tds-mds1 kernel: [22068.945517]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.042826] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.056208] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.317735] CPU 9 
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.326687] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.597364] 
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.606330] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.637545] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.667297] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.686459] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.707227] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.727859] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.756537] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.777083] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.797851] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.827856] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.847337] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.876383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.896952] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.917667] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.947698] Stack:
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.956763]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22151.977515] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.006591] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.028228] Call Trace:
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.037385]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.057917]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.086826]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.108384]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.137514]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.166899]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.188081]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.217134]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.238298]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.267281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.287270]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.307744]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.336577]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.367670]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.378160]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.406906]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.418062]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.446769]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.467177]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.486788] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.547761] Call Trace:
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.557193]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.577615]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.598084]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.628170]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.657072]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.678266]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.707506]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.728546]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.757506]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.778230]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.798206]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.826939]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.847477]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.868077]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.888186]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.916740]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.928039]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.956774]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:17:59 atlas-tds-mds1 kernel: [22152.977169]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:19:09 atlas-tds-mds1 kernel: [22223.005974] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 20:19:09 atlas-tds-mds1 kernel: [22223.033597] Lustre: Skipped 19 previous similar messages
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.073782] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.087540] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.358106] CPU 9 
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.359033] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.628979] 
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.638107] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.677589] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.699056] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.718225] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.738895] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.767829] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.788542] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.808958] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.838008] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.859673] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.878968] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.908112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.928706] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.949334] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.979409] Stack:
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22235.988441]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.009308] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.038464] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.068297] Call Trace:
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.070316]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.097791]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.118154]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.139419]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.168744]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.189614]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.218944]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.248174]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.268965]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.298204]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.318026]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.338430]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.358951]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.379416]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.407947]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.428318]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.439499]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.468253]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.488544]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.507946] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.569095] Call Trace:
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.578511]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.598872]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.619347]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.648892]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.669832]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.699116]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.728592]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.749318]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.778417]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.799136]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.819151]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.839482]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.868280]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.888795]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.909167]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.929472]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.949047]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.969472]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:19:23 atlas-tds-mds1 kernel: [22236.989586]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:20:03 atlas-tds-mds1 kernel: [22276.668997] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 20:20:03 atlas-tds-mds1 kernel: [22276.703895] Lustre: Skipped 20 previous similar messages
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.104730] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.118722] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.388781] CPU 9 
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.389877] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.660063] 
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.669035] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.710824] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.730028] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.749313] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.770161] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.798948] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.819503] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.840563] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.869294] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.890972] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.910322] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.939172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.960080] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22319.989013] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.019366] Stack:
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.020910]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.049216] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.070114] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.099809] Call Trace:
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.109081]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.129459]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.149695]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.179517]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.200365]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.229668]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.250794]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.279909]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.300927]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.329907]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.350116]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.370386]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.399206]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.419573]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.439755]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.460265]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.479782]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.500200]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.520618]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.540119] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.609233] Call Trace:
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.610970]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.639474]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.659759]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.681087]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.710427]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.739909]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.761090]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.790052]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.811143]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.840336]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.860377]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.880610]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.909599]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.930087]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.950128]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.970627]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22320.990115]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22321.010491]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:20:47 atlas-tds-mds1 kernel: [22321.030800]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:22:06 atlas-tds-mds1 kernel: [22399.542681] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 20:22:06 atlas-tds-mds1 kernel: [22399.579213] LustreError: Skipped 4 previous similar messages
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.135670] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.150221] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.420497] CPU 9 
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.421567] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.692012] 
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.700912] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.740398] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.761684] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.780997] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.801521] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.830640] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.851352] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.880304] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.901048] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.930846] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.950261] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.970863] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22403.991445] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.020516] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.042123] Stack:
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.051436]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.080337] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.101542] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.131260] Call Trace:
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.140565]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.160755]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.181341]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.211226]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.231945]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.261419]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.290813]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.311418]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.340831]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.361559]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.381583]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.402052]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.430936]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.451554]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.471607]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.492062]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.511647]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.540524]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.560867]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.571960] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.641491] Call Trace:
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.650954]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.671308]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.691967]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.721724]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.742387]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.771777]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.801446]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.822146]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.861713]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.881739]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.901729]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.922193]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.951105]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.971601]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22404.991613]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22405.011936]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22405.031628]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22405.052137]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:22:11 atlas-tds-mds1 kernel: [22405.080853]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.166603] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.181449] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.451567] CPU 9 
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.452654] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.722519] 
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.731549] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.762799] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.792246] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.811533] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.832303] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.853113] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.881756] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.902644] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.931602] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.953056] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.972352] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22487.993266] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.022186] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.042733] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.072984] Stack:
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.082053]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.102721] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.131909] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.153489] Call Trace:
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.162500]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.182826]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.203167]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.233012]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.262075]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.283200]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.312495]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.333213]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.362303]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.383388]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.403026]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.431801]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.452373]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.472927]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.492899]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.513205]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.532918]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.553292]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.581845]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.593084] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.662210] Call Trace:
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.671859]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.692084]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.712596]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.742645]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.763440]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.792546]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.813561]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.842516]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.863429]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.892532]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.912506]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.932769]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.953471]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22488.982161]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22489.002372]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22489.022732]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22489.042250]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22489.062614]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:23:35 atlas-tds-mds1 kernel: [22489.083144]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.197529] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.212796] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.474501] CPU 9 
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.483113] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.753784] 
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.762866] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.794206] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.823568] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.842837] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.863654] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.884171] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.913100] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.934012] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.954384] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22571.984394] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.003834] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.024467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.053474] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.074150] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.104263] Stack:
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.113294]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.134392] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.163386] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.185021] Call Trace:
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.194087]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.214620]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.243391]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.264849]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.304452]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.324445]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.353812]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.374699]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.403822]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.424514]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.444600]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.473119]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.493652]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.514027]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.534235]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.563114]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.574387]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.594693]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.623538]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.634565] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.703719] Call Trace:
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.713184]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.733607]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.754106]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.783985]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.804750]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.834068]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.855016]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.884400]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.913631]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.934519]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.954365]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22572.974723]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.003702]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.024062]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.044285]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.064765]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.084109]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.104426]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:24:59 atlas-tds-mds1 kernel: [22573.124896]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.228447] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.243902] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.505672] CPU 9 
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.514604] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.785311] 
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.794136] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.825219] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.854895] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.874152] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.894895] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.915311] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.944356] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.965138] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22655.985596] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.015531] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.034978] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.055503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.084411] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.104997] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.135210] Stack:
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.144218]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.164886] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.185784] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.215613] Call Trace:
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.224886]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.245238]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.265653]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.295307]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.316001]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.345438]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.374940]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.395691]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.424958]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.445858]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.465944]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.494390]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.515125]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.535673]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.555758]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.584468]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.595557]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.616046]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.644592]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.655704] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.724867] Call Trace:
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.734514]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.754875]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.775290]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.805327]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.826021]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.855308]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.884907]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.905496]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.934944]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.955688]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.975665]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22656.995851]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.025002]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.045335]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.065605]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.085874]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.105555]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.125889]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:26:23 atlas-tds-mds1 kernel: [22657.154775]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.259359] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.275394] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.545887] CPU 9 
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.546954] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.826341] 
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.835569] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.866863] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.896384] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.915604] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.936350] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.965502] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22739.986052] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.006687] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.035549] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.057025] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.076476] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.105518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.126109] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.146883] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.176882] Stack:
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.186135]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.206934] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.236055] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.257376] Call Trace:
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.266738]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.287257]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.315825]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.337445]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.366468]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.387286]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.416726]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.437448]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.466630]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.487281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.515856]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.535923]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.556643]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.577107]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.597147]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.626100]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.637408]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.666220]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.686515]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.705923] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.767122] Call Trace:
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.776598]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.797108]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.825810]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.847328]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.876618]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.897468]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.926986]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.947652]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.977043]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22740.997656]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.026078]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.046615]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.067069]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.095913]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.115928]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.136296]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.155937]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.176336]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:27:47 atlas-tds-mds1 kernel: [22741.196874]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.290264] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.306679] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.577080] CPU 9 
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.578353] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.848466] 
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.857443] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.896892] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.918212] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.937478] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.958066] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22823.987131] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.008009] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.036820] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.057364] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.087207] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.098235] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.127398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.147949] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.168387] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.198495] Stack:
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.207592]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.228295] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.257594] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.287481] Call Trace:
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.289206]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.317102]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.337684]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.367495]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.388356]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.417712]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.438883]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.467743]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.488762]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.517827]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.537737]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.558137]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.587041]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.607590]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.627786]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.648120]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.667654]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.688195]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.708450]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.728205] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.797044] Call Trace:
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.798954]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.827112]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.847513]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.877501]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.898346]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.927696]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.959264]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.977929]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22824.998976]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.028295]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.048353]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.068554]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.097618]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.117984]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.138150]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.158297]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.177808]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.198206]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:29:11 atlas-tds-mds1 kernel: [22825.218481]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:30:03 atlas-tds-mds1 kernel: [22876.892769] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 20:30:03 atlas-tds-mds1 kernel: [22876.917054] Lustre: Skipped 20 previous similar messages
Mar 31 20:30:03 atlas-tds-mds1 kernel: [22876.928148] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 20:30:03 atlas-tds-mds1 kernel: [22876.977324] Lustre: Skipped 19 previous similar messages
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.321163] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.338024] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.608512] CPU 9 
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.617536] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.888630] 
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.889842] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.928969] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.958620] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.969508] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22907.998686] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.019355] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.048271] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.069029] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.089391] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.119448] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.138915] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.159561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.188568] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.209252] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.239319] Stack:
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.248384]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.269113] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.298231] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.319852] Call Trace:
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.328981]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.349279]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.369800]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.399444]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.428627]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.449756]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.479239]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.499980]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.529231]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.550035]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.578362]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.598561]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.619401]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.639594]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.659757]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.688551]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.699643]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.728556]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.748956]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.768400] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.829414] Call Trace:
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.838772]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.859189]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.879774]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.909409]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.930237]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.959802]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22908.989194]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.009984]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.039234]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.059955]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.079762]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.100096]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.128915]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.149333]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.169484]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.189838]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.209496]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.229833]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:30:35 atlas-tds-mds1 kernel: [22909.258555]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.352054] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.369309] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.639397] CPU 9 
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.640409] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.910268] 
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.919324] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.950460] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.980138] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22991.999337] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.020115] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.040511] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.069619] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.090334] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.110926] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.140887] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.160374] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.181033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.210025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.230719] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.260889] Stack:
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.269932]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.290470] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.320239] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.350059] Call Trace:
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.351932]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.379557]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.400090]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.429959]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.450670]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.480130]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.501252]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.530240]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.551186]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.580179]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.600342]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.620718]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.641086]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.669998]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.690196]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.710396]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.730036]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.750379]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.770735]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.790016] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.850961] Call Trace:
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.860456]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.880657]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.901125]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.931318]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.960381]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22992.981497]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.010702]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.031407]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.060712]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.081556]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.109797]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.130202]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.150888]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.171292]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.191407]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.220007]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.231178]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.259937]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:31:59 atlas-tds-mds1 kernel: [22993.280069]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.382940] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.400650] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.671288] CPU 9 
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.680134] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.950631] 
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.951948] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23075.991003] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.012449] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.031428] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.052322] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.081121] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.102118] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.130946] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.151474] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.181317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.200910] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.221424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.242032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.270961] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.301237] Stack:
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.302693]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.330817] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.351905] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.381613] Call Trace:
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.390861]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.411227]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.431575]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.461428]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.482326]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.511844]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.532835]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.561857]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.582855]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.611852]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.631851]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.652413]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.681106]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.701709]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.721896]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.742208]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.761728]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.782069]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.802392]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.821909] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.890990] Call Trace:
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.892885]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.921030]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.941429]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.962817]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23076.992233]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.021445]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.042514]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.063086]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.092437]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.121726]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.141535]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.161919]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.182494]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.211334]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.231723]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.252025]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.271514]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.291739]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:33:23 atlas-tds-mds1 kernel: [23077.312058]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:34:36 atlas-tds-mds1 kernel: [23149.818710] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 20:34:36 atlas-tds-mds1 kernel: [23149.858816] LustreError: Skipped 4 previous similar messages
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.413819] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.431894] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.703087] CPU 9 
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.711993] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.982550] 
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23159.983823] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.022855] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.052421] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.063511] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.092386] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.113153] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.141999] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.162779] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.183445] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.213536] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.233223] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.253611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.282424] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.303174] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.333240] Stack:
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.342278]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.363044] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.392090] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.413570] Call Trace:
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.423040]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.443445]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.472304]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.493650]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.522921]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.544011]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.573299]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.594034]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.623490]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.652680]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.672371]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.692790]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.713502]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.742243]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.762469]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.782840]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.802327]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.822903]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.843322]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.862703] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.923756] Call Trace:
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.933226]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.953725]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23160.982406]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.004005]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.032754]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.054069]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.083402]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.104303]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.133372]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.154075]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.182401]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.202823]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.223526]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.244136]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.272435]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.292805]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.312489]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.332892]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:34:47 atlas-tds-mds1 kernel: [23161.353229]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23243.444692] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23243.463118] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23243.733380] CPU 9 
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23243.734437] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.004201] 
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.013259] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.044536] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.074282] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.093688] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.114247] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.134892] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.163915] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.184528] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.213495] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.235013] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.254448] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.283443] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.304115] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.324704] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.354703] Stack:
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.363775]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.384646] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.413783] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.435221] Call Trace:
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.444390]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.464905]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.493639]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.515183]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.544092]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.565235]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.594422]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.615297]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.644699]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.665468]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.693768]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.714107]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.734704]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.755101]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.783562]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.803729]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.814859]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.843654]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.874281]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.884670] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.953604] Call Trace:
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.955491]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23244.983653]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.004056]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.025475]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.054582]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.075534]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.105005]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.134061]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.155014]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.184100]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.204111]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.224643]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.245013]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.273927]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.294153]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.314332]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.334079]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.354155]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:36:11 atlas-tds-mds1 kernel: [23245.374715]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23327.475559] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23327.494361] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23327.765352] CPU 9 
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23327.774054] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.044987] 
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.046334] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.085508] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.115188] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.126220] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.155231] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.175876] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.204816] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.225371] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.246039] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.276185] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.295665] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.324682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.345240] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.365922] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.396080] Stack:
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.405077]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.425782] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.454909] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.476632] Call Trace:
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.485764]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.506176]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.534792]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.556238]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.585354]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.606325]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.635740]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.656498]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.685766]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.706712]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.734830]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.755049]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.775759]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.796322]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.816277]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.845045]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.856202]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.884902]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.905082]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.916293] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.985492] Call Trace:
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23328.995115]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.015502]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.035911]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.065497]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.086303]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.115741]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.136688]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.165763]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.186658]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.215775]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.235755]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.256274]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.285190]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.305693]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.325707]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.346086]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.365529]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.385848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:37:35 atlas-tds-mds1 kernel: [23329.406176]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23411.506420] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23411.525844] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23411.795968] CPU 9 
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23411.797134] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.066816] 
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.075898] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.106998] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.136465] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.147481] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.176295] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.196852] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.225915] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.246522] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.267317] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.296967] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.326977] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.346542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.367311] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.396026] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.417839] Stack:
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.426943]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.456015] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.476930] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.506809] Call Trace:
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.516092]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.536462]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.556889]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.586465]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.607374]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.636853]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.658027]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.686933]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.707672]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.736898]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.756847]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.777095]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.806341]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.826685]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.846761]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.867402]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.886946]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.907270]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.936164]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23412.947473] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.016579] Call Trace:
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.026206]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.046478]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.067017]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.096840]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.117732]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.147065]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.168123]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.196874]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.217799]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.246900]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.266898]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.287478]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.308060]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.336955]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.356957]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.377338]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.396741]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.416993]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:38:59 atlas-tds-mds1 kernel: [23413.437346]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23495.537275] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23495.557150] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23495.818662] CPU 9 
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23495.827446] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.098098] 
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.107197] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.138443] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.167983] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.187198] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.207952] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.228691] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.257619] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.278198] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.307244] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.328789] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.348174] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.368711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.397856] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.418654] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.448652] Stack:
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.457672]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.478593] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.507747] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.529136] Call Trace:
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.538305]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.558639]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.587354]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.608889]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.637952]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.659053]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.688356]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.709073]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.738220]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.758946]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.778867]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.807728]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.828277]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.848596]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.868773]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.897437]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.908598]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.937470]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.957879]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23496.968929] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.038274] Call Trace:
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.047733]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.068144]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.088507]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.118371]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.139452]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.168579]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.189448]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.218620]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.239439]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.268423]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.288576]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.308803]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.337664]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.358057]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.378260]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.398426]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.418091]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.438484]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:40:23 atlas-tds-mds1 kernel: [23497.458599]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:41:03 atlas-tds-mds1 kernel: [23537.514703] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 20:41:03 atlas-tds-mds1 kernel: [23537.516711] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 20:41:03 atlas-tds-mds1 kernel: [23537.516713] Lustre: Skipped 20 previous similar messages
Mar 31 20:41:03 atlas-tds-mds1 kernel: [23537.612865] Lustre: Skipped 22 previous similar messages
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23579.568125] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23579.588416] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23579.858719] CPU 9 
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23579.859874] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.130043] 
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.139223] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.178683] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.200112] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.219243] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.239844] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.268896] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.289676] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.318528] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.339318] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.369091] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.388754] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.409408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.430027] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.458761] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.489193] Stack:
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.490746]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.518810] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.540019] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.569828] Call Trace:
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.578851]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.599316]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.619776]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.649637]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.670343]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.699843]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.729269]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.750030]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.779344]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.800227]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.820143]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.848838]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.869384]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.889876]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.909866]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.930412]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.949918]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.970360]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23580.998999]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.010356] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.079413] Call Trace:
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.089020]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.109444]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.129635]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.159601]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.180548]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.209848]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.239285]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.260082]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.289378]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.310139]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.330271]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.350385]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.379533]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.400033]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.420103]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.448984]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.460289]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.480478]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:41:47 atlas-tds-mds1 kernel: [23581.509529]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23663.598969] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23663.611002] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23663.881014] CPU 9 
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23663.890041] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.160566] 
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.161775] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.200901] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.230631] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.250028] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.270636] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.291311] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.320019] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.340970] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.361291] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.391367] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.410789] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.439895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.460630] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.481214] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.511249] Stack:
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.520242]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.541029] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.570126] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.591831] Call Trace:
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.600953]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.621501]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.650160]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.671459]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.700770]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.721941]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.751166]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.771825]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.811829]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.831145]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.851154]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.871583]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.900527]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.921004]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.941039]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.961219]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23664.980701]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.001205]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.021655]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.041193] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.110292] Call Trace:
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.112174]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.140347]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.160979]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.190917]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.211489]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.240975]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.261951]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.291060]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.311904]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.340886]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.360802]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.381161]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.401693]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.430594]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.450854]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.471198]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.490842]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.511037]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:43:11 atlas-tds-mds1 kernel: [23665.531530]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23747.629808] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23747.642184] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23747.912105] CPU 9 
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23747.920977] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.191726] 
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.192946] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.232061] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.261634] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.272673] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.301577] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.322172] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.342598] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.371709] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.392310] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.422330] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.441772] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.462298] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.491373] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.511779] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.541959] Stack:
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.543609]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.571575] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.592736] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.622433] Call Trace:
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.631626]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.652029]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.672599]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.702297]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.723118]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.752617]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.781891]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.802689]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.831886]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.852810]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.872604]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.901396]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.922068]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.942497]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.962639]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23748.991396]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.002670]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.031469]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.051838]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.062924] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.132374] Call Trace:
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.141822]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.162211]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.182590]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.212428]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.233276]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.262459]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.291883]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.312641]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.342179]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.362655]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.382624]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.411559]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.431998]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.452623]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.472562]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.493064]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.512593]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.532860]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:44:35 atlas-tds-mds1 kernel: [23749.561608]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23831.660641] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23831.672916] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23831.943362] CPU 9 
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23831.952252] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.222758] 
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.224110] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.273611] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.293739] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.313007] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.333709] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.362658] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.383406] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.403874] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.433228] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.463037] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.482473] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.503105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.523976] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.552752] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.582939] Stack:
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.584410]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.612686] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.633641] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.663539] Call Trace:
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.672807]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.693355]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.713688]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.743585]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.764492]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.793829]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.823189]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.843951]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.873282]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.893965]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.914064]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.934171]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.963220]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23832.983630]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.003664]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.023964]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.043687]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.063826]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.092725]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.103993] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.173110] Call Trace:
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.182758]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.203200]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.223540]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.253581]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.274399]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.303796]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.333198]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.353985]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.374846]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.403994]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.423962]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.444303]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.473272]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.493631]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.513692]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.533988]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.553564]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.574045]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:45:59 atlas-tds-mds1 kernel: [23833.594369]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:47:06 atlas-tds-mds1 kernel: [23900.094053] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 20:47:06 atlas-tds-mds1 kernel: [23900.128456] LustreError: Skipped 4 previous similar messages
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23915.691469] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23915.703684] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23915.973976] CPU 9 
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23915.974987] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.244969] 
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.254106] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.285307] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.315045] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.334306] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.354977] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.384133] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.404470] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.425232] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.454344] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.484136] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.495356] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.524257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.544771] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.573824] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.595668] Stack:
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.604565]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.625430] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.654858] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.684585] Call Trace:
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.693851]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.714307]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.734726]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.764479]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.785208]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.814721]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.835833]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.865109]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.885590]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.914759]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.934904]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.955151]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23916.983964]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.004537]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.024592]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.044959]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.064618]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.085115]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.105298]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.124793] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.185859] Call Trace:
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.195151]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.215566]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.244268]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.265797]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.294688]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.315985]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.345247]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.365926]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.406094]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.425131]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.445093]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.465479]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.494550]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.515156]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.535139]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.555544]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.575218]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.595746]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:47:23 atlas-tds-mds1 kernel: [23917.624224]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [23999.722316] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [23999.734772] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.005104] CPU 9 
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.006257] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.276249] 
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.285379] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.316781] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.346320] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.365406] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.385980] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.406489] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.435640] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.456217] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.485087] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.506579] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.526275] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.555323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.576033] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.596636] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.626678] Stack:
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.635742]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.656717] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.685737] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.715719] Call Trace:
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.717575]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.745169]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.765780]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.795793]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.816577]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.845966]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.867066]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.896160]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.917129]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.946000]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.966020]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24000.986227]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.015286]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.035909]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.056050]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.076335]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.095966]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.116071]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.136596]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.156290] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.225375] Call Trace:
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.227452]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.255396]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.275818]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.297294]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.326573]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.355977]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.376903]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.406004]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.426936]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.455994]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.476049]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.496227]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.516942]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.545639]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.565943]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.586182]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.605822]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.626155]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:48:47 atlas-tds-mds1 kernel: [24001.646244]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24083.753414] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24083.766248] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.036633] CPU 9 
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.037788] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.307621] 
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.316702] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.347748] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.377274] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.396481] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.417564] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.446322] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.467150] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.487510] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.516755] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.538155] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.557704] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.586558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.607337] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.627884] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.658229] Stack:
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.667324]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.688458] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.717355] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.747165] Call Trace:
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.748886]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.776708]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.797069]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.826890]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.858279]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.878068]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.907283]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.928231]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.957343]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.978174]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24084.997999]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.026808]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.047407]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.068148]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.088052]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.116984]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.128260]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.156822]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.177215]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.196776] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.257796] Call Trace:
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.267079]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.287452]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.308014]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.337692]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.358667]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.388115]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.417468]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.438234]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.467596]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.488380]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.508152]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.536982]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.557520]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.577869]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.598091]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.626979]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.638056]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.658448]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:50:11 atlas-tds-mds1 kernel: [24085.686998]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:51:03 atlas-tds-mds1 kernel: [24137.778574] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 20:51:03 atlas-tds-mds1 kernel: [24137.779853] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 20:51:03 atlas-tds-mds1 kernel: [24137.779856] Lustre: Skipped 20 previous similar messages
Mar 31 20:51:03 atlas-tds-mds1 kernel: [24137.866657] Lustre: Skipped 19 previous similar messages
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24167.784505] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24167.797372] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.067765] CPU 9 
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.068850] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.338579] 
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.347592] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.379033] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.408767] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.428003] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.448571] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.477618] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.498230] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.518775] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.547678] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.569172] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.588592] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.609274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.638115] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.658839] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.689134] Stack:
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.698354]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.719146] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.748149] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.769591] Call Trace:
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.778987]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.799114]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.827835]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.849127]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.878597]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.899477]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.928789]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.949495]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.978649]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24168.999351]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.019290]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.048072]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.068497]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.089072]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.109092]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.129524]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.149012]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.169269]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.197882]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.209189] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.277907] Call Trace:
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.279682]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.308157]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.328614]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.358399]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.379294]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.408586]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.429646]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.458789]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.479531]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.508800]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.528630]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.549166]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.569516]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.598459]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.618527]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.639007]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.658496]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.678885]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:51:35 atlas-tds-mds1 kernel: [24169.699201]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24251.815588] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24251.828850] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.099181] CPU 9 
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.100416] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.379695] 
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.380902] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.420038] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.449719] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.460535] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.489575] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.510045] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.539175] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.559743] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.580675] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.610425] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.629950] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.650445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.679452] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.700121] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.730213] Stack:
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.739424]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.760133] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.789257] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.810995] Call Trace:
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.820116]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.840307]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.869103]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.890459]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.919529]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.940627]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.970250]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24252.990959]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.020279]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.040955]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.069160]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.089595]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.110147]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.130618]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.150625]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.179419]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.190798]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.219264]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.239655]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.259205] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.320247] Call Trace:
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.329694]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.350046]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.370597]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.400250]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.420988]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.450452]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.479924]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.500518]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.529749]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.550678]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.570646]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.591040]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.619779]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.640145]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.660487]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.680816]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.700346]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.720714]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:52:59 atlas-tds-mds1 kernel: [24253.741001]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24335.846665] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24335.859978] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.130488] CPU 9 
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.131493] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.401632] 
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.410787] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.450181] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.471543] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.491235] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.511933] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.541112] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.561114] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.581699] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.610816] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.640699] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.651715] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.680879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.701423] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.730574] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.760721] Stack:
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.762266]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.790579] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.811452] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.841309] Call Trace:
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.850591]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.871126]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.891374]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.921139]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.942051]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.971367]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24336.992269]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.021387]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.042257]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.071244]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.091234]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.111573]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.132150]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.160857]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.180913]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.201366]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.220946]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.241300]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.261620]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.281275] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.342265] Call Trace:
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.351477]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.371895]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.400798]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.432580]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.451726]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.481141]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.502094]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.531185]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.552091]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.581065]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.601076]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.621308]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.642007]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.670683]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.690841]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.711130]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.730711]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.750992]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:54:23 atlas-tds-mds1 kernel: [24337.771293]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24419.877735] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24419.891334] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.161360] CPU 9 
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.162544] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.432772] 
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.442066] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.481485] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.502828] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.522287] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.542865] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.571916] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.592692] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.613119] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.642165] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.672149] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.691570] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.712183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.732709] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.761826] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.783574] Stack:
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.792367]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.813358] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.842384] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.872255] Call Trace:
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.881717]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.902135]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.922702]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.952277]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24420.973004]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.002477]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.023676]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.052684]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.073697]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.102711]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.122502]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.142974]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.171983]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.192662]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.212590]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.232891]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.252477]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.273050]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.293510]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.312870] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.381817] Call Trace:
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.383723]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.412123]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.432611]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.462291]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.483205]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.512512]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.533715]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.562709]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.583735]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.612542]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.632515]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.652917]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.673393]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.702146]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.722217]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.742573]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.762215]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.782376]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:55:47 atlas-tds-mds1 kernel: [24421.803022]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24503.908799] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24503.922648] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.193123] CPU 9 
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.194123] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.472981] 
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.474221] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.513382] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.534688] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.553778] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.574274] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.603619] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.624266] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.653108] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.673672] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.703507] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.722916] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.743670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.764204] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.793361] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.823356] Stack:
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.824899]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.853341] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.883711] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.913568] Call Trace:
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.915470]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.943421]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.963682]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24504.993500]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.014546]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.043820]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.064703]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.093826]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.114821]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.143883]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.163827]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.184186]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.213141]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.233480]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.253735]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.274121]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.293759]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.314106]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.334558]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.354168] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.423126] Call Trace:
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.425036]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.453401]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.473802]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.503790]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.524508]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.553835]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.574901]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.603854]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.624939]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.654057]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.674011]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.694504]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.723403]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.744018]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.764105]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.784645]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.804106]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.824433]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:57:11 atlas-tds-mds1 kernel: [24505.844766]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24587.939856] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24587.953895] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.224409] CPU 9 
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.225562] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.495880] 
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.504754] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.544321] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.565786] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.584918] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.605475] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.634603] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.655039] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.675578] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.704374] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.725878] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.745292] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.774226] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.794836] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.815590] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.845804] Stack:
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.854663]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.875548] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.904666] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.926227] Call Trace:
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.935405]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.955630]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24588.984520]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.006102]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.035087]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.056227]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.085669]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.114954]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.135905]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.164766]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.184964]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.205160]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.225812]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.254598]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.274821]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.295173]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.314625]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.335051]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.355386]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.375028] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.436223] Call Trace:
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.445434]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.465882]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.494562]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.516230]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.545285]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.566445]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.595849]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.624962]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.645933]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.674986]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.695007]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.715251]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.736077]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.764808]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.785033]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.805382]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.824833]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.845262]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:58:35 atlas-tds-mds1 kernel: [24589.865813]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:59:36 atlas-tds-mds1 kernel: [24650.371321] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 20:59:36 atlas-tds-mds1 kernel: [24650.407576] LustreError: Skipped 4 previous similar messages
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24671.970906] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24671.985469] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.256881] CPU 9 
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.265833] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.536389] 
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.545423] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.576782] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.606227] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.625679] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.646529] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.675473] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.696293] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.716602] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.745721] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.767312] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.786659] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.815511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.835937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.856657] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.886863] Stack:
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.895968]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.916788] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.945983] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.967387] Call Trace:
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.976734]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24672.997014]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.025609]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.047278]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.076430]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.097453]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.126785]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.147421]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.176679]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.197492]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.225848]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.246178]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.266900]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.295708]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.315710]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.336121]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.347189]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.376201]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.396536]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.416141] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.477147] Call Trace:
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.486667]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.507163]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.535839]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.557324]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.586415]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.607317]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.637012]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.657670]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.687217]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.716281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.736077]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.756333]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.776814]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.797200]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.817327]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.846113]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.857365]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.886143]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 20:59:59 atlas-tds-mds1 kernel: [24673.906713]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:01:03 atlas-tds-mds1 kernel: [24738.084237] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) reconnecting
Mar 31 21:01:03 atlas-tds-mds1 kernel: [24738.110406] Lustre: Skipped 19 previous similar messages
Mar 31 21:01:03 atlas-tds-mds1 kernel: [24738.121311] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 21:01:04 atlas-tds-mds1 kernel: [24738.170929] Lustre: Skipped 20 previous similar messages
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.001951] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.016525] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.286798] CPU 9 
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.287922] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.557915] 
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.566886] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.598194] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.627782] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.646947] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.667597] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.688175] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.717379] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.738069] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.767224] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.788460] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.808077] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.837036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.857694] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.878338] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.908376] Stack:
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.917390]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.937982] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.967279] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.988836] Call Trace:
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24756.998039]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.018223]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.047122]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.068474]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.097578]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.118686]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.147904]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.168909]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.197935]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.218843]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.246992]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.267444]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.298441]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.317415]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.337614]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.357972]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.377497]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.397866]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.418186]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.437628] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.498636] Call Trace:
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.508075]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.528388]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.548612]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.578435]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.607730]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.628638]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.658140]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.678821]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.708154]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.728824]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.757194]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.777566]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.798143]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.818632]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.847229]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.867640]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.887258]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.907689]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:01:23 atlas-tds-mds1 kernel: [24757.927707]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.032989] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.047808] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.318138] CPU 9 
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.319282] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.589676] 
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.598770] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.638435] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.659673] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.678899] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.699597] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.728842] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.749345] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.778294] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.799069] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.828767] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.848141] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.868879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.889245] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.918325] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.948515] Stack:
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.950058]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.978170] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24840.999242] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.029132] Call Trace:
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.038225]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.058596]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.079014]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.108668]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.129607]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.159069]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.180092]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.209254]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.238728]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.259281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.279248]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.299567]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.328377]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.348861]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.368916]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.389349]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.408712]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.429127]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.449596]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.469107] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.530039] Call Trace:
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.539323]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.559806]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.588710]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.610103]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.639210]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.660279]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.689783]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.718895]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.739672]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.768863]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.788833]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.809239]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.829822]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.858724]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.878796]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.899110]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.918890]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.939197]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:02:47 atlas-tds-mds1 kernel: [24841.959486]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.064022] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.079176] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.349620] CPU 9 
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.350692] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.620381] 
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.629547] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.660834] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.690484] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.709753] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.730346] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.762080] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.779817] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.800503] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.829618] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.851136] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.870778] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.899423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.920073] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.940941] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.971034] Stack:
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24924.980000]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.000947] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.030159] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.059486] Call Trace:
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.061289]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.089772]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.110105]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.131437]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.160509]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.189970]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.210894]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.240061]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.260904]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.290169]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.310012]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.330564]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.350908]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.379838]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.400025]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.420263]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.439884]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.460114]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.480590]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.500013] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.561105] Call Trace:
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.570526]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.590921]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.619728]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.641248]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.670314]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.691226]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.720693]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.741429]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.770729]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.791441]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.819806]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.840244]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.860912]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.881440]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.909810]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.930206]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.941324]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.970095]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:04:11 atlas-tds-mds1 kernel: [24925.990641]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.095048] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.110489] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.380901] CPU 9 
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.381965] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.651746] 
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.660889] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.692257] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.721795] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.741204] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.761818] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.790715] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.811071] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.831881] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.860731] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.882218] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.901863] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.930920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.951578] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25008.972192] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.002080] Stack:
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.011176]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.031713] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.060980] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.082609] Call Trace:
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.091784]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.112212]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.140923]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.162607]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.191637]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.212822]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.242365]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.271263]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.292374]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.321481]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.341459]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.361923]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.382589]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.410946]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.431124]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.451565]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.470947]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.491372]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.511553]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.531213] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.592295] Call Trace:
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.601591]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.622117]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.642315]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.672224]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.692989]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.722390]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.751855]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.772592]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.801875]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.822724]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.842553]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.871317]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.891681]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.912258]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.932464]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.971662]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25009.982428]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25010.002783]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:05:35 atlas-tds-mds1 kernel: [25010.031562]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.126069] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.141724] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.412233] CPU 9 
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.413377] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.683558] 
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.692822] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.731931] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.753456] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.772696] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.793246] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.822443] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.843108] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.872074] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.892815] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.922681] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.942189] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.962810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25092.983625] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.012539] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.042816] Stack:
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.044358]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.072890] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.093667] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.123651] Call Trace:
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.132945]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.153098]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.173460]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.203265]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.223951]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.253283]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.282759]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.303334]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.332773]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.353499]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.373313]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.393659]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.422990]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.443133]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.463212]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.483513]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.503189]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.523552]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.552420]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.563571] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.632670] Call Trace:
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.642283]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.662733]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.683075]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.712777]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.733487]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.762941]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.784036]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.813165]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.834148]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.862983]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.882965]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.903407]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.924149]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.952981]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.973016]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25093.993189]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25094.012861]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25094.033288]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:06:59 atlas-tds-mds1 kernel: [25094.053634]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.157084] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.173231] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.443364] CPU 9 
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.444376] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.714577] 
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.723631] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.763261] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.784697] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.804132] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.824624] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.853508] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.874216] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.894763] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.923890] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.945323] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.964624] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25176.993606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.014295] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.034696] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.065051] Stack:
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.073928]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.094817] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.123810] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.153422] Call Trace:
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.155203]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.183644]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.203888]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.225335]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.254472]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.283880]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.304995]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.334096]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.365457]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.384478]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.404608]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.424967]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.453961]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.474289]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.494518]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.514806]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.534382]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.554559]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.574950]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.594657] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.663613] Call Trace:
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.665350]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.693688]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.714042]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.735431]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.764645]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.785491]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.815049]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.844147]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.864874]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.894060]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.913937]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.934311]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.954848]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.975416]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25177.995366]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25178.024163]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25178.035318]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25178.064196]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:08:23 atlas-tds-mds1 kernel: [25178.084510]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.188094] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.204551] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.474691] CPU 9 
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.475781] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.745484] 
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.754567] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.785832] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.815545] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.834745] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.855349] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.875872] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.905012] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.925559] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.946142] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.976147] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25260.995387] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.016138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.045015] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.065634] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.095634] Stack:
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.104659]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.125511] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.154674] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.176180] Call Trace:
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.185492]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.205845]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.226411]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.256122]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.285340]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.306373]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.335755]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.356534]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.385895]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.406558]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.426345]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.455016]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.475796]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.496300]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.524916]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.545343]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.556518]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.584919]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.605240]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.624903] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.685658] Call Trace:
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.695090]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.715693]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.736047]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.765720]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.786596]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.815937]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.845404]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.866174]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.895424]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.916154]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.936116]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.956589]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25261.985216]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.005807]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.025808]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.046413]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.065836]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.086473]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:09:47 atlas-tds-mds1 kernel: [25262.115048]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.219098] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.235654] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.497370] CPU 9 
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.506338] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.776823] 
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.786058] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.827801] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.847172] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.866516] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.887158] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.916146] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.936734] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.957399] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25344.986502] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.016366] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.027411] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.056387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.077099] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.106288] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.127928] Stack:
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.137031]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.166012] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.186754] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.217012] Call Trace:
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.219432]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.246470]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.266895]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.296689]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.317385]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.346711]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.367676]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.397063]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.417886]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.447109]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.466904]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.487299]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.516376]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.536758]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.556785]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.577122]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.596779]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.617143]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.637476]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.657130] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.726226] Call Trace:
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.727961]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.756443]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.776665]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.798104]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.827283]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.856900]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.877789]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.906949]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.927894]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.956888]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.976914]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25345.997100]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.017691]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.046522]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.066581]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.086965]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.106432]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.126804]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:11:11 atlas-tds-mds1 kernel: [25346.147167]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:11:39 atlas-tds-mds1 kernel: [25374.168998] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 21:11:39 atlas-tds-mds1 kernel: [25374.197654] Lustre: Skipped 20 previous similar messages
Mar 31 21:11:39 atlas-tds-mds1 kernel: [25374.216921] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) refused reconnection, still busy with 2 active RPCs
Mar 31 21:11:39 atlas-tds-mds1 kernel: [25374.257794] Lustre: Skipped 20 previous similar messages
Mar 31 21:12:06 atlas-tds-mds1 kernel: [25400.671534] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 21:12:06 atlas-tds-mds1 kernel: [25400.707298] LustreError: Skipped 4 previous similar messages
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.250097] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.267214] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.537083] CPU 9 
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.538099] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.808358] 
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.817600] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.848970] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.878571] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.897828] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.918676] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.947455] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.968056] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25428.988639] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.017634] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.039123] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.058720] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.087681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.108257] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.137313] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.158986] Stack:
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.168132]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.189021] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.218279] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.247346] Call Trace:
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.249106]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.277418]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.297844]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.327773]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.348394]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.377773]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.398906]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.428207]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.449108]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.477996]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.498042]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.518251]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.538604]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.567452]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.587698]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.608095]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.627704]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.648297]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.668583]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.688084] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.749217] Call Trace:
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.758663]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.778843]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.799172]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.829104]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.858071]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.889420]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.909090]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.938233]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.959070]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25429.988224]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.008257]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.028603]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.049192]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.077809]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.098061]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.118260]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.137880]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.158137]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:12:35 atlas-tds-mds1 kernel: [25430.178458]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.281091] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.298284] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.569175] CPU 9 
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.578024] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.848880] 
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.850105] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.889449] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.918905] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.929811] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.958746] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25512.979520] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.000160] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.029153] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.049692] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.079564] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.099006] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.119886] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.140122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.169262] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.199179] Stack:
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.200721]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.229032] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.249851] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.279674] Call Trace:
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.288867]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.309097]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.329694]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.359473]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.380358]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.409695]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.439119]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.460059]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.489360]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.510254]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.530186]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.558839]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.579440]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.599920]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.620484]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.648778]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.659979]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.688990]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.709230]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.728840] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.789914] Call Trace:
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.799296]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.819857]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.840198]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.869843]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.890553]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.920060]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.949470]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.970253]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25513.999547]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.020245]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.040048]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.060417]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.089201]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.109727]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.129955]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.150327]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.170003]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.190429]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:13:59 atlas-tds-mds1 kernel: [25514.219011]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.312080] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.329779] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.600087] CPU 9 
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.601097] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.870913] 
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.880003] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.911245] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.941008] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.960242] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25596.980950] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.009877] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.030577] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.051169] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.080091] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.101727] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.121394] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.150327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.170869] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.199971] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.221845] Stack:
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.230797]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.251708] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.280939] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.309970] Call Trace:
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.322433]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.340412]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.360775]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.390432]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.411340]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.440821]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.461876]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.491072]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.511958]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.540910]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.560957]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.581426]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.610504]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.631049]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.650900]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.671523]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.690918]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.711238]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.740153]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.751363] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.820586] Call Trace:
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.830211]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.850627]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.871128]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.900988]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.921553]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.951012]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25597.972043]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.001439]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.022281]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.051225]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.071198]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.091566]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.120468]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.140855]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.161082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.181249]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.200930]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.221435]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:15:23 atlas-tds-mds1 kernel: [25598.241868]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.343064] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.360894] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.631373] CPU 9 
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.632442] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.902545] 
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.911468] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.951392] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.972772] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25680.991934] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.012570] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.041763] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.062274] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.091421] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.112093] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.141787] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.161397] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.182029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.202630] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.231652] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.261899] Stack:
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.263381]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.291434] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.312372] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.342248] Call Trace:
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.351534]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.371859]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.392298]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.422299]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.443206]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.472473]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.501930]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.522686]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.551946]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.572694]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.592542]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.612993]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.641972]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.662509]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.682668]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.702969]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.722691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.751467]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.771633]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.782848] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.852237] Call Trace:
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.861743]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.882172]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.902425]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.932051]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.953030]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25681.982332]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.003507]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.032294]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.053240]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.082347]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.102503]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.122838]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.151608]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.172168]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.192344]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.212832]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.232380]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.252754]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:16:47 atlas-tds-mds1 kernel: [25682.273011]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.374042] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.392208] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.662755] CPU 9 
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.663760] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.943635] 
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.952694] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25764.983960] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.013551] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.032648] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.053426] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.082490] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.103260] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.123996] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.152889] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.174390] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.193807] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.222729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.243346] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.263949] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.294027] Stack:
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.303128]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.323806] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.353504] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.382578] Call Trace:
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.384434]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.412771]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.433201]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.462994]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.483770]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.513020]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.534170]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.563208]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.584213]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.613412]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.633271]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.653793]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.674436]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.703096]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.723252]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.743532]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.763361]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.783730]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.803760]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.823305] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.884393] Call Trace:
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.893746]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.914075]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.942826]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.964450]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25765.993484]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.014629]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.044004]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.064657]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.094014]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.114809]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.143116]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.163461]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.184092]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.204508]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.232909]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.253309]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.272923]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.293134]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:18:11 atlas-tds-mds1 kernel: [25766.313578]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.405017] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.423700] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.694052] CPU 9 
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.695122] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.965225] 
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25848.974326] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.013958] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.035228] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.054509] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.075302] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.104343] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.125355] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.154207] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.174896] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.204962] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.224213] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.244907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.273850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.294408] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.324461] Stack:
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.325998]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.354214] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.375132] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.404822] Call Trace:
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.414106]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.434509]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.455036]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.484885]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.505585]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.535062]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.564409]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.585346]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.614401]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.635118]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.655091]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.675436]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.704608]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.725041]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.745018]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.765347]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.784807]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.805336]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.825466]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.845137] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.914258] Call Trace:
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.916095]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.944262]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25849.964735]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.004981]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.025303]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.054688]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.075628]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.104717]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.125767]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.154837]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.174789]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.195341]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.224218]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.244648]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.264849]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.285334]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.304598]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.325189]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:19:35 atlas-tds-mds1 kernel: [25850.345655]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25932.435986] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25932.454862] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25932.725352] CPU 9 
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25932.726424] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25932.996330] 
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.005617] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.045050] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.066537] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.085858] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.106595] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.135500] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.156038] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.176574] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.205685] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.235538] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.246560] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.275339] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.296312] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.325192] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.346839] Stack:
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.355939]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.376889] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.406115] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.436007] Call Trace:
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.445296]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.465623]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.485914]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.515677]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.536441]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.565872]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.586981]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.616017]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.636974]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.666056]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.686286]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.706449]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.735563]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.755962]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.776104]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.796321]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.815953]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.836315]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.856793]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.876131] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.945439] Call Trace:
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.947278]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.975611]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25933.996177]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.025991]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.046595]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.076031]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.097122]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.126035]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.147022]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.176099]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.196230]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.216470]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.236967]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.266159]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.286148]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.306496]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.325957]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.346332]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:20:59 atlas-tds-mds1 kernel: [25934.366667]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26016.466951] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26016.486161] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26016.756793] CPU 9 
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26016.757855] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.027520] 
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.036530] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.067852] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.097477] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.116792] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.137480] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.166436] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.187178] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.207666] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.236634] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.258214] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.277774] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.306906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.327628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.356473] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.378157] Stack:
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.387387]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.408215] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.447239] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.477122] Call Trace:
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.479328]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.506896]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.527177]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.557093]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.578164]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.607356]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.636969]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.657701]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.678627]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.707556]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.727732]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.747924]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.777024]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.797556]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.817586]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.837946]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.857590]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.877811]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.898089]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.917750] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.986731] Call Trace:
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26017.988765]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.016758]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.037192]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.058690]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.087714]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.117158]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.138295]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.167147]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.188115]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.217326]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.237377]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.257753]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.278323]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.307185]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.327498]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.347656]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.367317]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.387545]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:22:23 atlas-tds-mds1 kernel: [26018.407694]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:22:33 atlas-tds-mds1 kernel: [26028.047836] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 21:22:33 atlas-tds-mds1 kernel: [26028.071035] Lustre: Skipped 20 previous similar messages
Mar 31 21:22:33 atlas-tds-mds1 kernel: [26028.081913] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 21:22:33 atlas-tds-mds1 kernel: [26028.131220] Lustre: Skipped 20 previous similar messages
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26100.497986] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26100.517541] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26100.787950] CPU 9 
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26100.789102] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.059239] 
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.068404] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.107942] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.129242] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.148468] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.169058] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.197911] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.218847] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.247705] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.268348] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.298288] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.317976] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.338518] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.359334] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.388184] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.418344] Stack:
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.419829]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.448144] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.469110] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.498759] Call Trace:
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.508000]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.528414]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.548908]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.578778]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.599502]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.628843]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.658254]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.679031]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.708340]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.729352]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.749095]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.777944]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.798527]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.819018]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.839069]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.859518]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.878904]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.899248]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.927988]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26101.939222] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.008165] Call Trace:
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.010050]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.038426]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.058822]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.088450]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.109362]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.138906]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.159983]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.188922]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.209797]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.239090]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.259007]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.279456]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.308308]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.328739]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.348903]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.369233]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.388915]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.409112]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:23:47 atlas-tds-mds1 kernel: [26102.429646]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:24:36 atlas-tds-mds1 kernel: [26150.925018] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 21:24:36 atlas-tds-mds1 kernel: [26150.957724] LustreError: Skipped 4 previous similar messages
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26184.529046] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26184.559262] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26184.820329] CPU 9 
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26184.829235] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.099532] 
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.100905] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.139762] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.169344] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.180289] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.209181] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.229963] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.259006] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.279608] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.300302] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.330183] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.349790] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.370250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.399232] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.419900] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.450138] Stack:
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.459362]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.480021] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.509085] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.530676] Call Trace:
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.539925]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.560254]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.580653]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.610456]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.639586]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.660663]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.690341]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.710842]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.740007]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.760889]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.780873]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.809423]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.830055]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.850661]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.870682]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.899644]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.910852]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.939565]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.959939]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26185.979373] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.040377] Call Trace:
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.049923]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.070119]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.090707]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.120480]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.149771]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.170850]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.200333]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.221193]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.250547]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.271361]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.299444]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.319993]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.340582]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.360896]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.389499]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.409736]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.420920]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.449877]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:25:11 atlas-tds-mds1 kernel: [26186.470358]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26268.560100] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26268.580244] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26268.850169] CPU 9 
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26268.851337] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.121198] 
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.130231] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.161384] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.191081] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.210364] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.230893] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.251626] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.280711] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.301460] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.330494] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.351890] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.371458] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.392057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.420964] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.441639] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.471716] Stack:
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.480949]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.501743] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.530615] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.552292] Call Trace:
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.561414]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.581757]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.610665]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.631943]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.660882]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.681978]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.711451]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.732136]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.761511]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.782274]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.810556]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.830974]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.863060]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.880981]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.901265]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.921527]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.940980]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.961571]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26269.981837]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.001189] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.062320] Call Trace:
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.071597]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.091920]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.120677]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.142135]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.171274]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.192463]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.221928]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.242622]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.271725]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.292498]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.320744]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.341101]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.361880]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.382322]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.410787]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.431159]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.442216]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.471185]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:26:35 atlas-tds-mds1 kernel: [26270.491497]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26352.591149] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26352.611532] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26352.881885] CPU 9 
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26352.883039] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.153189] 
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.162309] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.201980] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.223228] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.242545] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.271596] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.292207] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.312864] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.341862] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.362529] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.392261] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.411823] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.432440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.452961] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.482100] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.512233] Stack:
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.513776]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.542160] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.563252] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.592944] Call Trace:
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.602188]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.622361]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.642916]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.672969]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.693647]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.722968]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.752393]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.773239]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.802416]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.823122]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.843084]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.863486]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.892270]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.912753]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.932929]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.953297]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.972837]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26353.993160]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.021919]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.033318] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.101896] Call Trace:
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.103512]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.132134]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.152353]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.182345]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.202944]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.232391]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.253494]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.282727]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.303652]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.332743]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.352626]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.372987]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.393549]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.422396]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.442455]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.463066]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.482518]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.502848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:27:59 atlas-tds-mds1 kernel: [26354.523177]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26436.622190] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26436.642735] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26436.904171] CPU 9 
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26436.913143] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.183873] 
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.193046] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.224153] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.253636] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.272924] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.293678] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.324878] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.343577] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.364251] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.393135] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.415085] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.434427] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.463389] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.484109] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.513212] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.534822] Stack:
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.543924]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.573055] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.593976] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.623872] Call Trace:
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.633085]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.653292]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.673711]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.703455]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.724246]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.753651]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.774763]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.803912]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.824836]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.853973]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.873777]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.894481]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.923217]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.943720]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.963771]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26437.984117]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.003792]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.024167]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.044609]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.063976] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.133196] Call Trace:
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.134953]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.163339]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.183484]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.205056]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.234244]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.263684]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.284785]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.313887]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.334808]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.364012]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.383774]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.404103]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.424859]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.453529]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.473766]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.494104]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.513784]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.534148]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:29:23 atlas-tds-mds1 kernel: [26438.554755]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26520.653227] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26520.665412] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26520.944274] CPU 9 
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26520.945310] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.215149] 
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.224219] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.255333] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.284993] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.304200] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.324979] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.345412] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.374446] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.395197] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.415708] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.445772] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.465355] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.494347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.514932] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.535649] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.565652] Stack:
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.574571]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.595276] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.624529] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.646148] Call Trace:
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.655335]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.675904]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.704402]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.726088]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.755192]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.776292]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.805730]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.834859]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.855977]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.885035]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.905028]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.925225]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.945740]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.974670]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26521.994903]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.015077]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.034711]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.055156]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.075464]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.094726] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.155660] Call Trace:
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.165103]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.185526]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.205861]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.235728]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.264951]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.285963]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.315357]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.336009]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.365192]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.386270]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.405903]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.434664]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.455254]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.475611]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.496080]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.535160]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.545808]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.574739]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:30:47 atlas-tds-mds1 kernel: [26522.595124]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26604.684258] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26604.696438] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26604.966961] CPU 9 
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26604.975903] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.246396] 
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.255653] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.287002] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.316521] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.335795] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.356369] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.376945] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.405959] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.426635] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.447263] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.477047] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.496317] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.516867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.545959] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.566692] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.596906] Stack:
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.605795]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.626555] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.655803] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.677345] Call Trace:
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.686443]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.707037]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.735965]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.757262]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.786450]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.807430]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.836863]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.857789]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.886900]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.916132]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.935941]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.956384]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26605.977019]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.005815]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.025810]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.046144]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.065833]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.086168]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.106554]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.126017] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.187123] Call Trace:
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.196586]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.216951]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.237133]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.267025]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.287845]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.317219]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.346714]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.367546]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.396674]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.417777]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.445925]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.466343]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.486909]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.507376]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.536005]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.556337]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.576014]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.596368]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:32:11 atlas-tds-mds1 kernel: [26606.616755]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:32:33 atlas-tds-mds1 kernel: [26628.296214] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 21:32:33 atlas-tds-mds1 kernel: [26628.324683] Lustre: Skipped 19 previous similar messages
Mar 31 21:32:33 atlas-tds-mds1 kernel: [26628.344063] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 21:32:33 atlas-tds-mds1 kernel: [26628.385121] Lustre: Skipped 19 previous similar messages
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26688.715283] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26688.727655] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26688.997810] CPU 9 
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.006698] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.277176] 
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.278401] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.317599] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.347208] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.357972] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.387039] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.407626] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.428288] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.457360] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.478021] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.507490] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.527076] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.547519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.568164] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.597333] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.627328] Stack:
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.628813]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.656938] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.677736] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.707757] Call Trace:
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.716961]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.737192]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.757768]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.787620]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.808560]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.848512]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.868380]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.897428]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.918402]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.947442]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.967450]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26689.987811]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.008356]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.037070]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.057330]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.077688]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.097568]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.117773]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.138034]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.157551] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.218594] Call Trace:
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.227971]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.248182]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.277187]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.298737]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.327763]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.349034]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.378356]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.399291]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.428393]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.449241]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.477449]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.497758]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.518266]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.538378]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.558722]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.587269]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.598615]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.618911]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:33:35 atlas-tds-mds1 kernel: [26690.647698]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:34:36 atlas-tds-mds1 kernel: [26751.164255] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 21:34:36 atlas-tds-mds1 kernel: [26751.200493] LustreError: Skipped 3 previous similar messages
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26772.746293] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26772.758470] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.028745] CPU 9 
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.029742] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.299516] 
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.308566] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.339558] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.369271] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.388516] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.409255] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.438129] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.458934] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.479444] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.508587] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.529884] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.549661] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.578584] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.599551] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.628205] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.650077] Stack:
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.658980]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.679957] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.709024] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.738804] Call Trace:
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.740531]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.768721]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.788900]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.818717]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.839479]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.868730]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.889782]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.918920]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.939661]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.968951]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26773.988941]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.009342]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.029824]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.058591]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.078650]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.098803]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.118431]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.138799]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.159206]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.178616] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.239411] Call Trace:
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.248845]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.269261]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.289762]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.319613]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.340318]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.369688]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.399074]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.419872]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.449114]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.469872]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.489655]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.510027]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.538975]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.559336]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.579594]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.599907]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.619547]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.639900]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:34:59 atlas-tds-mds1 kernel: [26774.668826]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26856.777277] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26856.789386] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.059705] CPU 9 
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.060811] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.330801] 
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.339940] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.379600] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.401151] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.420184] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.440891] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.469850] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.490549] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.519716] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.540453] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.570262] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.589662] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.610262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.630933] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.659764] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.690062] Stack:
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.691606]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.719925] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.741046] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.770699] Call Trace:
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.779791]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.800160]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.820693]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.850529]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.871287]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.900794]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.930099]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.951012]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26857.980290]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.001052]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.020948]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.049849]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.070457]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.090791]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.111017]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.139707]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.150824]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.171145]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.200072]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.211194] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.280311] Call Trace:
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.289754]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.310369]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.330746]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.360605]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.381278]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.410811]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.440203]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.460822]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.481630]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.510793]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.530796]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.551193]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.580153]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.600703]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.620840]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.641138]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.660666]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.681102]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:36:23 atlas-tds-mds1 kernel: [26858.709911]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26940.808257] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26940.820529] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.082211] CPU 9 
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.091176] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.361654] 
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.370705] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.402019] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.431831] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.450940] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.471703] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.500753] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.521192] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.541899] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.570820] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.592446] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.611895] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.640837] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.661465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.681980] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.712231] Stack:
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.721281]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.741967] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.771266] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.792857] Call Trace:
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.802105]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.822536]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.851098]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.872638]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.901690]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.922726]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.952258]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26941.981333]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.002105]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.022802]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.051197]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.071593]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.092103]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.112438]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.141203]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.161479]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.181008]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.201464]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.221789]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.241217] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.302258] Call Trace:
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.311507]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.332086]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.352486]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.382297]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.411499]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.443077]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.462443]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.491512]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.512469]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.541726]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.561645]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.582019]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.602801]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.631609]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.651770]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.672185]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.691674]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.712262]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:37:47 atlas-tds-mds1 kernel: [26942.732481]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27024.839232] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27024.851969] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.122336] CPU 9 
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.123437] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.393594] 
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.402816] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.442224] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.463527] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.482983] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.503474] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.532249] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.552831] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.573649] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.602492] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.624102] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.643730] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.672771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.693262] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.722120] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.744005] Stack:
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.753092]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.782296] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.803253] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.832909] Call Trace:
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.842207]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.862605]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.882987]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.912796]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.933706]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.963021]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27025.984340]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.013216]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.042713]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.063614]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.083277]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.103746]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.132756]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.153469]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.173672]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.202512]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.213747]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.242536]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.262917]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.282483] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.343331] Call Trace:
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.352591]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.373195]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.393820]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.423363]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.452845]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.473919]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.503192]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.523910]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.553224]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.573927]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.593911]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.622520]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.643119]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.663445]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.683557]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.703853]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.723455]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.743789]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:39:11 atlas-tds-mds1 kernel: [27026.772538]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27108.870203] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27108.883303] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.153715] CPU 9 
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.154997] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.433322] 
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.434428] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.473727] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.495150] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.514283] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.543400] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.563988] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.584662] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.613806] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.634400] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.663975] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.683631] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.704178] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.733437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.754045] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.784040] Stack:
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.785579]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.813884] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.835083] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.864595] Call Trace:
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.884392]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.903703]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.924299]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.954112]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27109.975277]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.004481]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.025645]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.054570]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.075437]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.104724]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.124749]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.145023]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.173705]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.194240]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.214349]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.234588]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.254063]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.274419]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.294780]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.314410] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.375488] Call Trace:
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.384965]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.405143]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.433912]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.455381]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.484507]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.505548]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.534945]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.555782]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.584718]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.605426]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.633796]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.654040]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.674723]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.695078]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.715293]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.744018]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.755099]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.784081]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:40:35 atlas-tds-mds1 kernel: [27110.804441]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27192.901169] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27192.914478] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.175994] CPU 9 
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.185088] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.455718] 
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.464777] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.495939] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.525635] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.544725] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.565388] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.586058] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.615121] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.635636] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.655987] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.686024] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.705323] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.726138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.754951] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.775529] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.805723] Stack:
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.814730]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.835512] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.864737] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.886171] Call Trace:
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.895392]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.915806]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.936200]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.965936]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27193.986872]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.015990]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.045269]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.065999]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.095303]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.116016]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.136030]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.156457]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.185358]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.205638]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.225892]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.246197]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.265695]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.286080]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.306404]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.326109] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.386779] Call Trace:
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.396189]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.416490]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.445023]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.466517]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.495811]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.516862]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.546653]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.575627]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.596649]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.625862]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.645700]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.666052]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.686543]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.715319]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.735508]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.755877]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.775345]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.796047]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:41:59 atlas-tds-mds1 kernel: [27194.816774]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27276.932131] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27276.945884] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.216202] CPU 9 
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.217215] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.497031] 
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.506124] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.537588] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.567064] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.586166] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.606912] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.627387] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.656574] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.677215] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.706226] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.728047] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.747401] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.776209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.796981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.826034] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.847722] Stack:
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.857008]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.886086] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.907153] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.936865] Call Trace:
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.946133]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.966482]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27277.986928]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.016881]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.037656]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.066904]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.088016]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.116960]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.138073]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.166976]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.186772]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.207173]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.236241]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.256870]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.276828]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.297250]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.316698]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.337138]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.357428]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.377066] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.437800] Call Trace:
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.447217]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.467574]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.487836]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.517821]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.546748]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.567829]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.597314]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.618166]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.647332]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.668003]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.696408]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.716950]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.737430]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.758062]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.786476]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.806782]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.817858]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.846661]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:43:23 atlas-tds-mds1 kernel: [27278.866998]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:43:33 atlas-tds-mds1 kernel: [27288.879821] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 21:43:33 atlas-tds-mds1 kernel: [27288.897060] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 21:43:33 atlas-tds-mds1 kernel: [27288.897063] Lustre: Skipped 20 previous similar messages
Mar 31 21:43:33 atlas-tds-mds1 kernel: [27288.970618] Lustre: Skipped 22 previous similar messages
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27360.963089] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27360.977188] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.247541] CPU 9 
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.248546] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.518671] 
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.527803] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.567231] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.588739] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.608070] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.628443] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.657631] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.678263] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.698778] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.727886] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.757690] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.777330] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.797924] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.818425] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.847571] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.869317] Stack:
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.878298]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.907520] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.928531] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.958198] Call Trace:
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.967447]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27361.987879]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.008024]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.038008]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.058931]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.088306]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.109396]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.138457]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.159369]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.188465]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.208253]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.228697]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.257747]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.278093]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.298136]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.318502]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.337938]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.358340]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.378665]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.398315] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.467558] Call Trace:
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.469315]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.497872]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.518048]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.548041]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.568800]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.598196]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.619408]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.648410]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.669493]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.698427]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.718537]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.738639]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.767748]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.788256]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.808318]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.828844]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.848326]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.868841]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:44:47 atlas-tds-mds1 kernel: [27362.889155]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27444.994043] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.008521] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.278868] CPU 9 
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.280026] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.549810] 
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.558940] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.590138] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.620059] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.639287] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.659885] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.688855] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.709627] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.730123] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.759024] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.780530] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.799906] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.829087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.849739] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.870103] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.900487] Stack:
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.909566]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.938689] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.959636] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.989524] Call Trace:
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27445.998749]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.019151]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.039515]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.069183]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.090069]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.119387]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.140277]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.169396]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.190312]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.219442]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.239443]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.259586]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.288848]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.309372]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.329540]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.349780]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.369260]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.389671]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.410268]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.429620] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.490561] Call Trace:
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.499860]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.520223]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.548915]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.570446]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.599743]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.620628]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.650248]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.670998]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.700141]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.729401]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.749238]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.769659]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.790340]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.819404]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.839483]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.859626]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.879277]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.899511]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:46:11 atlas-tds-mds1 kernel: [27446.919888]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:47:06 atlas-tds-mds1 kernel: [27501.446492] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 21:47:06 atlas-tds-mds1 kernel: [27501.479809] LustreError: Skipped 4 previous similar messages
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.024993] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.039808] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.309944] CPU 9 
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.311094] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.581091] 
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.590022] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.621282] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.651011] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.670224] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.690826] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.711310] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.740561] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.761041] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.790297] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.822203] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.831496] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.860546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.881395] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.910025] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.940369] Stack:
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.941821]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.970130] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27529.991153] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.020942] Call Trace:
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.030228]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.050484]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.070952]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.100832]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.121693]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.151203]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.180639]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.201410]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.230641]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.251472]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.271381]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.300225]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.320764]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.341291]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.361293]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.390152]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.401380]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.421622]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.450403]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.461711] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.530768] Call Trace:
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.540365]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.560817]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.581145]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.611014]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.631883]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.661040]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.682211]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.711072]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.731948]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.761059]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.781059]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.801452]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.830497]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.850985]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.871108]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.891505]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.911035]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.931362]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:47:35 atlas-tds-mds1 kernel: [27530.951709]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.055939] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.070945] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.341436] CPU 9 
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.342498] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.612744] 
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.621805] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.652818] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.682501] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.701984] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.722856] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.751424] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.771976] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.792772] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.821667] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.842912] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.862418] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.891416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.912022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.932567] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.962801] Stack:
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.971900]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27613.992856] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.022065] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.051284] Call Trace:
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.053197]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.081375]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.101708]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.123170]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.152359]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.182039]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.203124]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.232199]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.253098]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.282182]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.302161]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.322559]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.351468]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.371851]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.392022]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.412580]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.432062]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.452588]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.473043]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.492573] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.561693] Call Trace:
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.563528]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.591894]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.612296]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.642170]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.662837]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.692130]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.713516]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.742566]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.763476]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.792366]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.812572]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.832763]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.861877]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.882409]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.902599]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.923073]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.942584]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.962981]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:48:59 atlas-tds-mds1 kernel: [27614.983249]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.086881] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.113130] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.374032] CPU 9 
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.382938] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.653613] 
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.662438] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.693826] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.723425] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.742698] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.763292] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.783945] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.812931] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.833641] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.862550] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.883958] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.903552] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.924048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.953196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27697.973733] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.003924] Stack:
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.013004]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.033857] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.063030] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.084476] Call Trace:
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.093591]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.113958]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.142687]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.164176]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.193264]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.214203]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.243873]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.264691]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.293864]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.314606]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.343181]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.363620]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.384113]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.412966]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.432953]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.453372]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.472769]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.493197]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.513391]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.532971] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.593821] Call Trace:
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.603242]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.623614]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.644209]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.674009]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.694731]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.724397]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.753495]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.774431]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.803700]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.824565]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.852924]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.873188]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.893893]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.914293]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.934508]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.963395]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27698.974285]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27699.003178]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:50:23 atlas-tds-mds1 kernel: [27699.023573]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.117821] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.133764] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.404462] CPU 9 
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.413286] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.675492] 
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.684487] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.723977] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.745270] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.764542] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.785067] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.814048] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.834831] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.855426] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.884440] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.914257] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.933908] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.954497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27781.975019] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.003918] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.025834] Stack:
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.034898]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.063926] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.084847] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.114476] Call Trace:
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.116353]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.144174]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.164703]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.194552]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.215131]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.244594]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.265785]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.294979]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.315816]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.344982]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.364945]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.385154]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.424733]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.444492]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.464707]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.485076]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.504463]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.525065]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.545205]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.564833] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.626015] Call Trace:
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.635103]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.655560]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.684354]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.705871]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.734996]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.756049]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.785337]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.805992]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.835358]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.855921]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.875819]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.904490]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.925179]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.945525]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.965675]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27782.994491]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27783.005701]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27783.034288]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:51:47 atlas-tds-mds1 kernel: [27783.054739]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.148765] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.165060] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.426700] CPU 9 
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.435618] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.706069] 
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.715292] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.746424] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.776091] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.795293] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.815704] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.836402] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.865337] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.885947] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.906728] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.936561] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.956086] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27865.976641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.005786] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.026348] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.056519] Stack:
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.065569]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.086494] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.115777] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.145244] Call Trace:
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.147011]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.175459]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.195873]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.225881]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.246611]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.275970]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.297156]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.326182]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.346987]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.376273]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.396305]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.416665]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.445610]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.465985]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.485994]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.506361]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.525940]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.546358]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.566660]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.586330] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.647456] Call Trace:
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.656691]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.685483]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.705833]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.727108]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.756509]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.785909]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.806966]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.835903]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.856865]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.885919]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.905907]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.926348]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.947044]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.975539]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27866.995794]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27867.016150]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27867.035786]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27867.056204]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:53:11 atlas-tds-mds1 kernel: [27867.076772]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:53:33 atlas-tds-mds1 kernel: [27889.106690] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 21:53:33 atlas-tds-mds1 kernel: [27889.134396] Lustre: Skipped 17 previous similar messages
Mar 31 21:53:33 atlas-tds-mds1 kernel: [27889.140083] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 21:53:33 atlas-tds-mds1 kernel: [27889.140086] Lustre: Skipped 19 previous similar messages
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.179707] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.196242] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.466878] CPU 9 
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.467941] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.746374] 
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.747665] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.796569] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.817910] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.837144] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.857680] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.886793] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.907395] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.936604] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.957214] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27949.987012] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.006673] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.027225] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.048013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.077062] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.107104] Stack:
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.108642]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.136871] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.157958] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.187590] Call Trace:
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.197033]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.217413]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.237773]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.267598]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.288387]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.317738]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.347178]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.367935]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.397211]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.417941]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.437736]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.458249]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.487199]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.507636]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.527781]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.548187]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.567804]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.588151]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.616920]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.628122] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.697339] Call Trace:
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.706942]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.727333]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.747740]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.777704]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.798590]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.827799]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.857219]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.877987]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.898674]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.927965]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.947946]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.968033]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27950.997030]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.017450]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.037611]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.057836]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.077475]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.098325]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:54:35 atlas-tds-mds1 kernel: [27951.118504]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.210644] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.227680] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.497806] CPU 9 
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.498778] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.768955] 
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.778047] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.809137] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.838875] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.858132] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.878849] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.907757] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.928343] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.949240] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.978127] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28033.999706] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.019013] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.048267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.068725] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.089241] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.119480] Stack:
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.128576]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.149311] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.178415] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.207880] Call Trace:
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.209735]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.238080]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.258498]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.288282]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.309083]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.338686]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.359740]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.388903]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.418332]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.439082]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.458914]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.479421]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.508402]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.528726]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.548932]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.569274]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.588771]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.609145]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.629497]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.649133] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.710121] Call Trace:
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.719314]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.739739]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.768462]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.789800]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.819231]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.840059]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.869619]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.898744]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.919704]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.959366]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.978551]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28034.998925]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.019457]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.048388]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.068463]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.088812]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.108409]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.128819]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:55:59 atlas-tds-mds1 kernel: [28035.149160]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.241578] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.258933] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.529346] CPU 9 
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.530371] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.800254] 
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.809388] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.840686] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.870435] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.889659] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.910463] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.939385] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.960028] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28117.980567] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.009271] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.030706] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.050359] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.079248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.099867] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.120584] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.150779] Stack:
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.159887]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.180443] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.209510] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.231136] Call Trace:
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.240436]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.269201]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.289598]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.311175]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.340161]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.369618]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.390573]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.419639]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.440703]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.469650]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.489686]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.510090]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.530570]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.559297]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.579292]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.599673]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.610905]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.639670]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.660066]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.679476] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.740456] Call Trace:
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.749925]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.770225]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.790754]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.820584]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.841305]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.870723]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.900284]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.920820]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.949990]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.970966]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28118.990946]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.019719]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.040301]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.060605]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.080731]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.109476]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.120640]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.141084]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:57:23 atlas-tds-mds1 kernel: [28119.169701]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.272508] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.290324] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.560584] CPU 9 
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.561589] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.831400] 
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.840316] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.871728] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.901310] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.920525] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.941109] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.961640] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28201.990758] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.011502] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.040407] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.062041] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.081523] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.110611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.131306] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.151707] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.181933] Stack:
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.191000]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.211751] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.240895] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.262544] Call Trace:
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.271555]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.292071]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.320752]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.342133]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.371314]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.402925]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.421669]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.442590]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.471719]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.492363]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.520847]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.541200]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.561571]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.590592]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.610825]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.631172]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.642183]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.670835]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.691209]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.710644] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.771811] Call Trace:
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.781515]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.801658]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.822057]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.851721]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.872508]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.901842]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.931276]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.951882]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28202.981303]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.002051]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.021867]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.042370]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.071175]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.091366]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.111584]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.131886]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.151212]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.171936]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 21:58:47 atlas-tds-mds1 kernel: [28203.192236]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 21:59:36 atlas-tds-mds1 kernel: [28251.699454] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 21:59:36 atlas-tds-mds1 kernel: [28251.739476] LustreError: Skipped 4 previous similar messages
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.303435] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.321560] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.591863] CPU 9 
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.593023] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.871788] 
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.873094] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.912173] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.933542] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.952566] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28285.981661] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.002377] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.022797] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.052028] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.072633] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.102449] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.121936] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.142459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.163406] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.192294] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.222385] Stack:
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.223844]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.252157] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.273377] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.302899] Call Trace:
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.311952]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.332572]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.352901]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.382628]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.403454]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.433135]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.462374]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.483263]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.512480]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.533352]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.553126]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.581905]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.602641]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.623092]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.642917]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.663329]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.683034]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.703366]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.731955]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.743197] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.812376] Call Trace:
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.822014]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.842390]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.862751]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.892583]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.913314]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.942843]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.963969]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28286.992821]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.013836]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.043005]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.062794]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.083378]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.112078]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.132695]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.152683]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.173256]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.192704]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.213052]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:00:11 atlas-tds-mds1 kernel: [28287.233578]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.334359] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.352905] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.623211] CPU 9 
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.624219] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.902898] 
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.904194] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.943351] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.964666] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28369.983920] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.012965] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.033722] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.054298] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.083362] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.103986] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.133785] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.153416] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.173928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.194640] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.223283] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.253436] Stack:
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.255059]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.283275] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.304265] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.333867] Call Trace:
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.343087]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.363499]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.383926]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.414067]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.435010]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.463924]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.484948]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.514160]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.543545]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.564307]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.584278]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.613205]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.633633]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.654283]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.674337]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.694931]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.714364]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.734740]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.763238]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.774510] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.843638] Call Trace:
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.853245]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.873693]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.894083]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.923887]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.944820]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.974169]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28370.995090]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.024135]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.045000]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.074307]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.094483]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.114881]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.143576]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.163967]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.184204]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.204382]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.223973]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.244572]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:01:35 atlas-tds-mds1 kernel: [28371.264857]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.365279] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.383996] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.654695] CPU 9 
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.655700] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.926009] 
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.934930] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.974462] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28453.995730] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.014985] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.035695] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.064463] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.085056] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.105715] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.134928] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.164688] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.184309] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.204894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.225594] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.254510] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.284732] Stack:
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.286276]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.314358] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.335463] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.365356] Call Trace:
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.374644]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.394780]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.415172]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.444997]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.465774]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.495191]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.516288]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.545410]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.566473]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.595443]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.615504]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.636060]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.664776]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.685146]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.705304]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.725739]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.745486]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.765827]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.794536]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.805643] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.874830] Call Trace:
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.876689]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.904775]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.925064]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.954995]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28454.975604]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.015583]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.036169]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.065019]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.086170]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.115223]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.135250]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.155481]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.176123]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.204945]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.225083]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.245476]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.264926]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.285144]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:02:59 atlas-tds-mds1 kernel: [28455.305526]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:04:09 atlas-tds-mds1 kernel: [28525.318526] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 22:04:09 atlas-tds-mds1 kernel: [28525.341388] Lustre: Skipped 20 previous similar messages
Mar 31 22:04:09 atlas-tds-mds1 kernel: [28525.352251] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) refused reconnection, still busy with 2 active RPCs
Mar 31 22:04:09 atlas-tds-mds1 kernel: [28525.401585] Lustre: Skipped 22 previous similar messages
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.396196] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.415384] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.685437] CPU 9 
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.686546] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.956631] 
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.965717] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28537.996958] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.026484] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.045709] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.066498] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.095542] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.116156] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.136848] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.165834] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.187310] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.206743] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.235654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.256203] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.276952] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.306788] Stack:
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.315982]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.336736] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.365991] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.387469] Call Trace:
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.396635]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.417191]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.445788]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.467307]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.496540]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.517609]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.546877]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.576166]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.596886]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.617710]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.646000]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.666164]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.686928]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.707522]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.727424]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.756215]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.767571]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.796238]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.816605]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.836030] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.896975] Call Trace:
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.906437]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.927059]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.947170]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.977023]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28538.997711]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.027454]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.056678]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.077475]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.106687]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.127465]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.147400]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.176189]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.196871]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.217078]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.237536]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.257432]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.277149]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.297623]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:04:23 atlas-tds-mds1 kernel: [28539.326206]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.427110] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.446795] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.716685] CPU 9 
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.717724] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.987958] 
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28621.997026] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.028122] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.057798] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.077007] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.097712] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.118279] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.147469] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.168252] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.197147] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.218580] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.238210] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.267274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.287893] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.316921] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.338833] Stack:
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.347869]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.387503] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.407945] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.437567] Call Trace:
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.439569]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.467175]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.487617]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.517565]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.538180]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.567592]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.588748]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.617621]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.638808]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.667829]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.687857]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.708244]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.728721]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.757532]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.777691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.798054]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.817570]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.838126]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.858264]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.877886] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.939093] Call Trace:
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.948456]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.968792]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28622.997443]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.018989]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.048035]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.068884]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.098544]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.127656]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.148420]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.169212]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.197520]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.217903]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.238529]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.267276]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.287330]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.307747]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.318896]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.347738]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:05:47 atlas-tds-mds1 kernel: [28623.368068]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28705.458020] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28705.478081] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28705.748370] CPU 9 
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28705.749381] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.019487] 
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.028526] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.068183] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.089533] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.108732] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.129265] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.158185] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.178845] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.199378] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.228403] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.249853] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.269531] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.298218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.318820] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.339672] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.369935] Stack:
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.378823]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.399726] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.428847] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.458314] Call Trace:
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.460232]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.488338]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.508686]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.530434]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.559453]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.588740]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.609662]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.638949]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.659852]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.689015]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.708960]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.729182]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.749829]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.778606]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.798585]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.818994]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.829887]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.858883]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.879235]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.898646] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.959716] Call Trace:
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.969170]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28706.989516]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.009641]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.039692]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.060170]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.089884]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.119134]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.139714]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.169110]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.189874]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.209913]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.230178]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.259002]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.279529]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.299558]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.320172]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.339590]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.360062]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:07:11 atlas-tds-mds1 kernel: [28707.388833]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28789.488928] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28789.509201] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28789.780936] CPU 9 
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28789.789909] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.060746] 
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.069823] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.109416] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.130940] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.150045] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.170656] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.199648] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.220114] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.240793] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.269716] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.291187] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.310799] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.339779] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.360514] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.389573] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.411434] Stack:
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.420485]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.441054] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.470544] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.500363] Call Trace:
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.509606]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.530081]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.550639]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.580204]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.601163]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.630461]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.651483]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.680645]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.710055]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.730829]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.750651]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.771274]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.800004]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.820477]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.840848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.861159]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.880558]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.901100]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.930007]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28790.941096] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.010405] Call Trace:
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.019787]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.040431]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.060764]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.090647]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.111480]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.140978]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.170232]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.191050]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.220427]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.241247]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.261156]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.289910]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.310524]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.331029]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.351051]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.379904]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.391058]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.411404]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:08:35 atlas-tds-mds1 kernel: [28791.440171]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28873.519846] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28873.531941] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28873.811021] CPU 9 
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28873.812026] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.090927] 
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.092192] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.131151] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.152438] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.171706] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.200955] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.221716] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.242267] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.271009] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.291797] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.321373] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.340993] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.361762] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.382363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.411234] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.441433] Stack:
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.443090]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.471378] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.492493] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.521867] Call Trace:
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.531266]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.551855]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.572352]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.602160]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.622891]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.652298]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.681518]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.702355]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.731529]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.752335]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.772417]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.792552]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.821447]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.841950]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.862290]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.882457]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.902178]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.922423]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.951239]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28874.962468] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.041093] Call Trace:
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.042974]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.071165]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.091585]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.113087]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.142255]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.171551]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.192481]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.221567]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.242481]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.271576]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.291546]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.311971]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.332318]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.361225]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.381388]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.401791]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.421296]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.441851]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:09:59 atlas-tds-mds1 kernel: [28875.462198]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28957.550761] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28957.563101] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28957.841941] CPU 9 
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28957.842961] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.112950] 
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.122015] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.153300] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.182832] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.202049] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.222817] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.243660] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.272704] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.293227] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.322275] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.343753] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.363067] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.392165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.412710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.433455] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.463648] Stack:
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.472585]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.493364] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.522638] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.544074] Call Trace:
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.553416]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.573724]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.602444]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.623893]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.653045]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.682631]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.703585]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.732651]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.753762]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.782897]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.802696]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.823073]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.843670]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.872521]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.892717]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.912939]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.932712]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.952979]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.973437]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28958.993143] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.053896] Call Trace:
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.063391]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.083813]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.112438]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.133975]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.163183]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.184121]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.213595]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.234276]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.263613]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.284621]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.312573]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.332926]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.353471]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.373906]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.394033]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.422904]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.434036]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.462764]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:11:23 atlas-tds-mds1 kernel: [28959.483273]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:12:06 atlas-tds-mds1 kernel: [29001.999004] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 22:12:06 atlas-tds-mds1 kernel: [29002.038807] LustreError: Skipped 4 previous similar messages
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29041.581674] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29041.593700] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29041.863972] CPU 9 
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29041.872837] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.143518] 
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.144813] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.184027] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.205128] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.224320] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.244843] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.273940] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.294835] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.323585] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.344176] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.384448] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.394879] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.423713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.444569] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.473458] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.495297] Stack:
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.504253]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.525070] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.554253] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.584104] Call Trace:
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.586140]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.613713]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.634157]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.663933]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.684672]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.714144]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.735193]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.764202]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.785230]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.814188]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.834323]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.854555]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.875206]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.904034]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.924075]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.944452]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.964154]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29042.984519]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.004667]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.024270] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.085143] Call Trace:
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.094438]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.115122]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.143715]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.165308]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.194537]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.215349]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.244753]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.265589]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.294717]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.315460]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.343781]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.364026]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.384744]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.405452]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.433873]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.454223]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.465271]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.494321]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:12:47 atlas-tds-mds1 kernel: [29043.514453]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:14:09 atlas-tds-mds1 kernel: [29125.543017] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 22:14:09 atlas-tds-mds1 kernel: [29125.565770] Lustre: Skipped 19 previous similar messages
Mar 31 22:14:09 atlas-tds-mds1 kernel: [29125.585145] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) refused reconnection, still busy with 2 active RPCs
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612584] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612585] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612618] CPU 9 
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612619] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612644] 
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612646] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612648] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612651] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612652] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612653] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612654] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612656] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612657] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612658] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612660] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612661] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612664] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612665] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612666] Stack:
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612667]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612669] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612672] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612675] Call Trace:
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612704]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612725]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612746]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612759]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612780]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612801]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612810]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612830]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612853]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612864]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612874]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612900]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612923]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612927]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612949]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612951]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612972]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612994]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612996]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.612997] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613009] Call Trace:
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613029]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613050]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613070]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613080]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613101]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613121]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613130]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613151]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613172]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613181]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613191]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613210]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613232]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613235]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613257]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613259]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613280]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613302]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29125.613304]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:14:11 atlas-tds-mds1 kernel: [29127.576391] Lustre: Skipped 19 previous similar messages
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29209.643490] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29209.655729] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29209.926239] CPU 9 
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29209.927301] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.197241] 
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.206308] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.237465] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.267140] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.286175] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.306761] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.327417] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.356437] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.377137] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.406048] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.427608] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.447241] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.476036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.496850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.517384] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.547586] Stack:
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.556654]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.577519] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.606677] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.636702] Call Trace:
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.638547]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.666402]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.686701]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.716562]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.737462]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.766995]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.788035]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.817018]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.837855]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.866808]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.886759]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.907186]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.927837]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.956676]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.976889]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29210.997070]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.016680]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.037232]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.057423]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.076897] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.138192] Call Trace:
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.147698]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.167838]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.196576]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.218130]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.247141]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.268218]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.297683]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.326773]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.347537]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.376778]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.396794]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.417031]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.437557]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.457942]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.486479]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.506867]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.517910]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.546674]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:15:35 atlas-tds-mds1 kernel: [29211.567083]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29293.674394] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29293.687054] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29293.948721] CPU 9 
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29293.957663] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.228087] 
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.229363] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.268678] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.297866] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.317294] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.338134] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.358650] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.387714] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.408533] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.437318] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.458815] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.478434] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.507393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.528092] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.548654] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.579014] Stack:
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.588113]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.608683] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.637961] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.667424] Call Trace:
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.669189]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.697450]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.717817]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.739336]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.768418]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.789354]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.818790]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.847909]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.879480]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.898487]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.918426]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.939025]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.967718]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29294.988165]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.008745]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.028684]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.048163]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.068548]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.089051]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.108354] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.169488] Call Trace:
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.178728]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.199216]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.227791]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.249289]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.278247]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.299482]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.328953]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.349487]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.379063]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.408114]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.427951]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.448096]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.468675]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.488996]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.509303]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.537904]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.549215]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.577755]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:16:59 atlas-tds-mds1 kernel: [29295.598173]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29377.705295] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29377.718362] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29377.988605] CPU 9 
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29377.989697] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.259901] 
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.268953] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.300229] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.329908] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.348968] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.369973] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.398589] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.419190] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.439727] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.468658] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.490274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.509797] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.538838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.559420] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.579941] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.610211] Stack:
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.619080]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.639933] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.669089] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.690669] Call Trace:
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.699975]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.720330]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.748964]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.770445]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.799586]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.820489]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.849937]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.870922]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.899963]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.929280]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.949256]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.969831]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29378.990116]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.019047]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.039027]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.059440]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.079141]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.099533]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.119791]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.139571] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.200534] Call Trace:
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.209878]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.230205]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.250368]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.280287]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.300842]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.330475]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.359710]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.380418]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.409751]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.430478]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.450603]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.479233]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.499784]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.520370]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.540682]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.569034]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.580355]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.609058]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:18:23 atlas-tds-mds1 kernel: [29379.629446]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29461.736193] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29461.749678] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.020180] CPU 9 
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.021180] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.290998] 
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.300228] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.339845] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.361428] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.380452] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.401188] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.430353] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.450887] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.479946] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.500524] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.530346] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.549930] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.570576] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.591247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.620404] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.650455] Stack:
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.652098]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.680070] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.701072] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.730904] Call Trace:
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.740063]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.760616]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.780852]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.810842]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.831618]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.861039]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.890475]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.911340]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.932198]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.961233]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29462.981402]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.001587]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.030772]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.051101]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.071309]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.100147]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.111112]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.131619]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.160174]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.171520] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.240636] Call Trace:
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.250218]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.270815]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.291256]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.320827]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.341586]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.370880]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.392028]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.421084]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.441965]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.471267]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.491231]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.511436]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.540367]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.560934]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.581162]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.601472]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.621141]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.641504]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:19:47 atlas-tds-mds1 kernel: [29463.661888]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29545.767088] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29545.781003] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.051458] CPU 9 
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.052523] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.322437] 
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.331597] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.362816] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.392329] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.411585] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.432324] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.461223] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.481897] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.502580] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.531430] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.552995] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.572582] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.601498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.622036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.642583] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.672830] Stack:
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.681904]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.702837] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.731952] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.761321] Call Trace:
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.763010]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.791414]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.811750]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.833230]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.862165]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.883369]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.912704]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.941801]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.962755]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29546.991783]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.011638]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.032058]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.052574]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.081462]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.101489]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.121888]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.133082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.161725]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.182304]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.201727] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.262873] Call Trace:
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.272316]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.292674]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.313088]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.342896]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.372166]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.393076]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.422576]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.443304]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.472429]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.503611]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.521870]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.542246]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.562681]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.583089]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.611734]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.632088]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.651651]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.672104]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:21:11 atlas-tds-mds1 kernel: [29547.692426]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29629.797981] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29629.812454] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.082369] CPU 9 
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.083389] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.353707] 
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.362660] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.394092] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.423804] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.443112] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.463718] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.492543] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.513251] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.533874] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.562760] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.584666] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.603842] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.632830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.653531] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.674068] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.704092] Stack:
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.713181]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.733901] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.762986] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.784861] Call Trace:
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.793948]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.814071]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.842912]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.864536]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.893512]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.914644]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.944142]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.973098]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29630.993879]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.014705]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.043153]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.063337]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.084186]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.112739]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.132996]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.153340]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.172804]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.193222]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.213664]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.233282] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.294403] Call Trace:
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.303644]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.324048]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.344163]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.374195]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.403253]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.424205]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.453647]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.474326]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.503753]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.524302]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.544184]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.564446]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.593391]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.613804]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.634119]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.654405]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.673919]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.694431]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:22:35 atlas-tds-mds1 kernel: [29631.723208]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29713.828870] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29713.843584] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.114047] CPU 9 
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.115143] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.385403] 
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.394549] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.433987] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.455296] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.474561] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.495100] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.524025] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.544600] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.565275] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.594294] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.624219] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.635331] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.664284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.684902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.705491] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.735568] Stack:
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.744480]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.765465] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.794467] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.823956] Call Trace:
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.825883]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.854173]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.874557]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.904406]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.925302]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.965074]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29714.985343]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.014486]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.035162]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.064441]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.084413]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.104658]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.125038]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.154117]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.174114]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.194518]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.205645]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.234292]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.254708]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.274296] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.335265] Call Trace:
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.344752]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.364963]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.385432]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.415349]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.444588]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.465705]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.495034]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.515716]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.544806]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.565544]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.585513]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.614266]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.635200]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.655529]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.675691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.704480]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.715601]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.744288]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:23:59 atlas-tds-mds1 kernel: [29715.764697]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:24:36 atlas-tds-mds1 kernel: [29752.251531] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 22:24:36 atlas-tds-mds1 kernel: [29752.288254] LustreError: Skipped 4 previous similar messages
Mar 31 22:25:03 atlas-tds-mds1 kernel: [29779.430082] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 22:25:03 atlas-tds-mds1 kernel: [29779.458765] Lustre: Skipped 20 previous similar messages
Mar 31 22:25:03 atlas-tds-mds1 kernel: [29779.469607] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 22:25:03 atlas-tds-mds1 kernel: [29779.519168] Lustre: Skipped 20 previous similar messages
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29797.859758] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29797.874868] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.145322] CPU 9 
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.146335] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.425084] 
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.426324] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.465277] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.486575] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.506027] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.526560] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.555709] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.576836] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.605389] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.626167] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.655968] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.675407] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.696110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.716664] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.745411] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.775620] Stack:
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.777157]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.805393] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.826760] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.856413] Call Trace:
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.865530]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.886123]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.906289]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.936044]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.957232]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29798.986652]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.016079]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.036635]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.065913]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.086646]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.106787]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.135381]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.155935]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.176475]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.196634]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.217031]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.236430]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.256726]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.285552]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.296664] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.365818] Call Trace:
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.375457]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.395814]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.416224]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.446007]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.466822]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.496213]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.517342]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.546270]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.567239]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.596466]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.616138]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.636492]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.656898]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.685826]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.705988]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.726340]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.746001]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.766539]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:25:23 atlas-tds-mds1 kernel: [29799.786855]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29881.890643] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29881.906169] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.176674] CPU 9 
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.177743] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.456732] 
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.457960] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.497158] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.518193] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.537515] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.566658] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.587262] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.607557] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.636507] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.657275] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.687058] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.706630] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.727416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.747827] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.776902] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.807063] Stack:
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.808748]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.836902] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.857789] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.887679] Call Trace:
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.896784]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.917139]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.937535]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.967517]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29882.988219]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.017568]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.038571]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.067763]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.088450]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.117589]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.137592]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.158080]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.186889]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.207252]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.227631]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.247943]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.267448]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.288197]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.316692]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.327982] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.396965] Call Trace:
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.398807]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.426952]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.447351]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.477154]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.497903]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.527190]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.548298]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.577387]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.598402]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.627470]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.647441]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.667953]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.688444]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.717213]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.737462]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.757955]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.777482]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.797947]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:26:47 atlas-tds-mds1 kernel: [29883.818373]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29965.921525] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29965.937471] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.207960] CPU 9 
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.209027] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.479189] 
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.488283] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.527667] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.549273] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.568282] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.589012] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.617902] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.638512] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.659440] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.688331] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.718131] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.729295] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.758176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.778721] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.807874] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.829471] Stack:
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.838476]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.859195] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.888427] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.917853] Call Trace:
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.919749]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.947900]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.968247]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29966.989755]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.018862]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.048316]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.069450]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.098729]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.119829]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.148715]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.168681]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.188922]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.217962]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.238460]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.258430]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.278936]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.298415]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.318936]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.339305]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.359025] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.428062] Call Trace:
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.429908]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.458047]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.489004]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.509387]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.538488]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.559597]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.589063]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.609735]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.639090]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.659623]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.688174]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.708358]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.728970]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.749559]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.769434]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.798192]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.809530]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.829848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:28:11 atlas-tds-mds1 kernel: [29967.858411]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30049.952405] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30049.968975] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.230369] CPU 9 
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.239257] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.509609] 
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.510833] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.549985] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.579594] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.590449] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.619441] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.640299] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.669221] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.689853] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.710325] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.740358] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.759639] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.780294] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.809185] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.829890] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.860371] Stack:
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.869273]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.889909] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.919132] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.940825] Call Trace:
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.949761]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.970296]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30050.990521]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.020378]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.041303]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.070558]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.099997]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.120704]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.150030]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.170721]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.190778]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.219484]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.240148]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.260374]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.280599]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.300764]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.320268]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.340732]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.369545]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.380583] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.449924] Call Trace:
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.459404]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.479804]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.500322]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.530162]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.551046]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.580380]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.609781]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.630565]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.651402]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.680584]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.700543]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.720971]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.749712]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.770250]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.790241]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.810741]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.830130]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.850561]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:29:35 atlas-tds-mds1 kernel: [30051.870869]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30133.983282] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.000256] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.270407] CPU 9 
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.271641] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.541511] 
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.550508] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.581588] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.611412] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.630624] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.651074] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.671674] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.700751] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.721461] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.750348] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.772060] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.791336] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.820475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.841074] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.861590] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.892043] Stack:
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.900834]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.932191] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.951157] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.981039] Call Trace:
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30134.982965]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.010471]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.030906]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.060857]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.081505]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.111052]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.132196]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.161071]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.182055]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.211099]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.231137]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.251578]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.272044]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.300777]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.320764]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.341167]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.360597]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.381008]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.401574]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.420784] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.481807] Call Trace:
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.491252]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.511598]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.531981]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.561683]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.582476]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.611857]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.641571]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.662049]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.691325]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.712082]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.732050]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.752250]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.781156]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.801518]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.821702]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.842199]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.861717]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.882173]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:30:59 atlas-tds-mds1 kernel: [30135.902502]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.014153] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.031576] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.301701] CPU 9 
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.302791] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.572865] 
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.581758] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.613334] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.642910] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.661985] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.682727] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.711857] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.732508] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.753217] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.782093] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.803496] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.822941] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.851903] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.872323] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.893288] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.923219] Stack:
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.932100]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.953028] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30218.982089] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.003820] Call Trace:
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.012871]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.033209]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.062177]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.083608]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.112588]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.133770]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.163119]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.192227]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.213136]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.242243]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.262026]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.282452]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.303215]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.331894]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.352179]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.372511]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.391959]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.412362]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.432866]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.452296] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.513085] Call Trace:
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.522545]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.542910]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.563611]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.593244]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.613969]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.643615]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.673106]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.693691]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.722996]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.743994]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.772061]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.792284]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.813042]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.833342]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.853569]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.882073]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.893590]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.922115]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:32:23 atlas-tds-mds1 kernel: [30219.942683]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.045021] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.062749] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.334379] CPU 9 
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.343344] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.614169] 
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.623295] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.654350] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.684031] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.703362] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.724045] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.752966] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.773708] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.794241] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.822959] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.844474] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.863970] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.884421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.913436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.934158] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.964342] Stack:
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.973377]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30302.994205] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.023240] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.044972] Call Trace:
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.054074]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.074441]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.103122]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.124576]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.153932]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.174822]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.204268]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.224964]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.254305]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.283549]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.303820]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.324044]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.344491]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.373430]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.393391]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.413817]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.433442]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.453801]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.474359]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.493600] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.554793] Call Trace:
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.564080]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.584548]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.604790]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.634779]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.663929]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.685146]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.714302]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.734991]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.764315]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.785013]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.805054]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.833569]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.854322]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.874708]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.894733]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.923483]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.934681]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.954938]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:33:47 atlas-tds-mds1 kernel: [30303.983809]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:35:03 atlas-tds-mds1 kernel: [30379.657256] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 22:35:03 atlas-tds-mds1 kernel: [30379.682391] Lustre: Skipped 19 previous similar messages
Mar 31 22:35:03 atlas-tds-mds1 kernel: [30379.701596] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 22:35:03 atlas-tds-mds1 kernel: [30379.742564] Lustre: Skipped 19 previous similar messages
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.075887] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.094147] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.364501] CPU 9 
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.365501] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.644561] 
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.645789] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.684966] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.714608] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.725593] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.754481] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.775073] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.795690] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.824651] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.845299] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.875268] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.894475] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.915263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.944297] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.964745] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.994963] Stack:
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30386.996623]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.024776] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.045967] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.075861] Call Trace:
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.084954]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.105318]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.125811]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.155509]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.176364]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.205848]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.235422]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.256079]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.285233]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.305929]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.325865]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.354650]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.375265]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.395731]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.426312]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.444926]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.456090]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.484776]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.505092]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.524711] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.585834] Call Trace:
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.595333]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.615607]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.635869]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.665727]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.686657]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.715932]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.745393]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.766260]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.795540]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.816179]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.835996]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.864658]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.885272]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.905600]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.925650]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.946037]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.965671]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30387.986001]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:35:11 atlas-tds-mds1 kernel: [30388.014933]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.106751] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.125497] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.395796] CPU 9 
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.396804] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.666608] 
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.675696] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.707060] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.736692] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.755740] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.776492] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.797093] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.825954] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.846539] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.867078] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.896936] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.916542] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.945677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.966347] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30470.986892] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.016935] Stack:
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.026005]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.046825] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.076029] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.098083] Call Trace:
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.107145]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.127273]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.155916]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.177422]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.206469]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.227812]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.256905]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.277557]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.306933]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.336141]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.356162]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.376402]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.397074]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.425776]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.446039]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.466497]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.485833]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.506309]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.526638]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.546250] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.607175] Call Trace:
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.616425]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.637056]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.657181]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.687247]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.716292]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.737511]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.766889]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.787440]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.816740]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.837465]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.865968]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.886182]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.906941]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.927263]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.947443]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.976220]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30471.987454]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30472.016110]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:36:35 atlas-tds-mds1 kernel: [30472.036408]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:37:06 atlas-tds-mds1 kernel: [30502.527028] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 22:37:06 atlas-tds-mds1 kernel: [30502.567789] LustreError: Skipped 4 previous similar messages
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.137614] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.156823] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.427128] CPU 9 
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.428126] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.698400] 
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.707564] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.747082] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.768319] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.787621] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.818756] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.837288] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.857975] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.886918] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.907465] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.937292] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.956905] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.977534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30554.998246] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.027372] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.057303] Stack:
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.058846]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.086971] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.108071] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.137946] Call Trace:
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.147215]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.167579]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.187953]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.217590]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.238525]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.267989]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.297439]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.318238]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.338946]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.368006]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.388047]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.408477]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.437543]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.457913]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.478114]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.498413]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.517867]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.538264]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.558723]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.578238] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.647338] Call Trace:
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.649073]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.677377]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.697785]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.727623]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.748332]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.777800]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.798709]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.827816]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.848737]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.877833]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.897862]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.918065]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.938575]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.967526]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30555.987717]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30556.008088]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30556.027767]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30556.047946]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:37:59 atlas-tds-mds1 kernel: [30556.068456]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.168473] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.187915] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.458358] CPU 9 
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.459366] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.729052] 
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.738124] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.769408] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.798918] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.818197] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.838988] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.859608] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.888778] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.909285] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.938383] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.959986] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30638.979605] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.008425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.029209] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.049749] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.079761] Stack:
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.088840]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.109605] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.138698] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.160220] Call Trace:
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.169450]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.190010]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.218711]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.240369]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.269439]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.298914]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.320230]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.348959]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.370019]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.399190]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.419340]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.439530]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.468421]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.488753]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.509006]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.529346]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.548820]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.569201]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.589675]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.609193] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.670269] Call Trace:
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.679558]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.699931]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.728652]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.750009]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.779049]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.800037]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.829311]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.850072]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.879120]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.899918]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.919661]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.939839]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.968812]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30639.989371]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30640.020080]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30640.038762]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30640.050290]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30640.079137]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:39:23 atlas-tds-mds1 kernel: [30640.099461]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.199331] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.219228] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.480826] CPU 9 
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.489840] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.760394] 
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.769658] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.800710] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.830396] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.849592] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.870214] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.899440] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.920230] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.940844] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.969729] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30722.991168] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.010741] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.039702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.060281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.080819] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.110882] Stack:
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.119929]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.140668] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.169740] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.191480] Call Trace:
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.200547]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.220854]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.249623]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.271115]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.300226]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.321271]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.350434]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.371420]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.400806]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.421562]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.449902]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.470267]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.491007]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.519712]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.539762]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.560106]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.571301]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.599969]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.620136]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.631396] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.700557] Call Trace:
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.710159]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.730553]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.750930]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.780760]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.801407]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.830656]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.851835]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.880612]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.901549]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.930597]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.950620]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.971144]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30723.991433]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.020308]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.040544]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.060859]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.080272]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.100733]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:40:47 atlas-tds-mds1 kernel: [30724.120866]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.230187] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.250723] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.521217] CPU 9 
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.522302] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.792342] 
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.801472] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.841313] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.862469] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.881722] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.910761] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.931540] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.952061] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30806.980987] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.001732] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.031580] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.050829] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.071620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.092082] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.121266] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.151603] Stack:
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.153220]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.181280] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.202321] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.232034] Call Trace:
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.241281]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.261626]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.282194]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.312058]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.332897]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.362261]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.391536]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.412286]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.452221]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.471985]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.492162]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.512376]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.541585]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.561809]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.582009]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.602598]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.621995]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.642172]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.671068]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.682200] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.751468] Call Trace:
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.761090]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.781319]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.801696]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.831695]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.852471]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.882030]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.903092]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.932467]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.961568]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30807.982506]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.002477]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.031215]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.051807]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.072311]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.092337]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.112705]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.132574]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.161246]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:42:11 atlas-tds-mds1 kernel: [30808.181650]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.261042] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.273287] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.552105] CPU 9 
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.553125] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.823124] 
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.832030] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.863489] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.893010] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.912449] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.933038] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.962078] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30890.982670] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.003211] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.032123] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.053745] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.073381] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.102116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.122900] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.143460] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.173536] Stack:
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.182621]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.203415] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.232758] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.262212] Call Trace:
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.264053]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.292425]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.312844]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.342625]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.363622]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.393001]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.413815]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.442821]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.463670]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.492880]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.512941]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.533106]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.553821]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.582717]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.602742]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.623285]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.642735]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.663120]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.683633]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.702948] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.764264] Call Trace:
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.773344]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.793865]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.822639]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.844238]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.873424]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.902805]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.923809]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.953021]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30891.973973]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.003266]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.023270]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.043600]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.072514]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.092912]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.113082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.133482]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.153111]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.173338]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:43:35 atlas-tds-mds1 kernel: [30892.194013]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.291894] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.304044] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.573794] CPU 9 
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.574861] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.853721] 
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.855092] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.903370] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.924730] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.943959] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.964614] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30974.993402] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.014180] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.034678] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.063825] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.085389] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.104756] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.133673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.154378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.174953] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.205912] Stack:
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.214638]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.235632] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.264526] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.294073] Call Trace:
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.295957]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.323724]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.344155]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.374114]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.395039]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.424349]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.445378]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.474353]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.495254]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.524565]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.544328]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.564893]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.593823]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.614163]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.634423]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.654580]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.674053]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.694450]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.714743]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.734440] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.795703] Call Trace:
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.804616]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.824987]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.853950]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.875370]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.904526]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.925699]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.954952]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30975.975560]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.005197]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.034208]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.054138]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.074534]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.095096]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.124010]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.144047]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.164437]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.183907]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.204282]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:44:59 atlas-tds-mds1 kernel: [30976.224450]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:46:03 atlas-tds-mds1 kernel: [31040.248767] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) reconnecting
Mar 31 22:46:03 atlas-tds-mds1 kernel: [31040.278535] Lustre: Skipped 20 previous similar messages
Mar 31 22:46:03 atlas-tds-mds1 kernel: [31040.297755] Lustre: atlastds-MDT0000: Client 6a5c48ea-8b4f-d046-431e-52c0fb4076be (at 2@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 22:46:03 atlas-tds-mds1 kernel: [31040.338706] Lustre: Skipped 22 previous similar messages
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.322745] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.334989] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.605281] CPU 9 
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.606281] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.876182] 
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.885065] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.924882] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.946133] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.965445] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31058.986104] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.015106] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.035639] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.056193] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.085314] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.106542] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.126506] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.155140] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.175715] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.204996] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.226809] Stack:
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.235782]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.264741] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.285716] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.315555] Call Trace:
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.317269]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.345251]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.365536]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.395469]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.416357]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.445808]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.466669]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.496005]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.525244]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.546020]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.565839]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.586354]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.615354]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.635891]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.656070]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.676619]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.696139]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.716297]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.745137]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.756239] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.825365] Call Trace:
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.834969]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.855424]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.875802]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.905868]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.926559]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.956059]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31059.995888]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.016025]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.045491]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.066218]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.086262]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.106516]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.135372]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.155931]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.176082]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.196253]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.215750]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.236105]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:46:23 atlas-tds-mds1 kernel: [31060.256672]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.353593] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.365715] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.627420] CPU 9 
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.636349] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.907026] 
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.916106] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.947392] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.977087] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31142.996165] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.016736] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.037281] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.066201] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.086913] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.107271] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.137166] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.156753] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.177444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.206219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.227000] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.257013] Stack:
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.266065]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.286842] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.316085] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.337692] Call Trace:
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.347044]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.367307]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.387711]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.417733]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.446750]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.467779]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.497154]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.517707]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.546927]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.567726]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.587554]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.608313]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.636838]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.657136]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.677397]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.697605]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.717195]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.737517]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.766256]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.777338] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.846248] Call Trace:
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.848189]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.876535]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.896873]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.926720]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.947521]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.976929]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31143.998132]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.027132]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.047864]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.076968]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.097116]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.117544]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.146407]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.166792]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.186998]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.207349]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.227027]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.247380]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:47:47 atlas-tds-mds1 kernel: [31144.267772]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.384447] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.397051] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.667484] CPU 9 
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.668494] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.938353] 
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.947430] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31226.978712] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.008401] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.027638] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.048202] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.068847] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.097866] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.118297] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.139013] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.168653] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.188097] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.208755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.237976] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.258513] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.288391] Stack:
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.297625]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.318361] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.347576] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.369082] Call Trace:
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.378175]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.398672]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.438001]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.459149]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.488025]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.509113]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.538599]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.559283]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.588667]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.609295]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.637745]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.658116]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.678486]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.698997]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.727527]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.747754]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.758981]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.787946]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.808307]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.827901] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.888767] Call Trace:
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.898189]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.918538]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.938972]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.968750]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31227.989669]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.018961]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.048437]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.069313]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.098809]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.119325]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.147957]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.168146]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.188709]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.209215]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.237775]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.258026]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.269262]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.297964]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:49:11 atlas-tds-mds1 kernel: [31228.318349]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:49:36 atlas-tds-mds1 kernel: [31252.802640] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 22:49:36 atlas-tds-mds1 kernel: [31252.837501] LustreError: Skipped 4 previous similar messages
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.415302] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.428541] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.698643] CPU 9 
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.699803] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.969640] 
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31310.978695] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.009885] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.039549] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.058972] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.079704] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.108602] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.129215] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.149972] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.178885] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.200403] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.219970] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.248842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.269451] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.298651] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.320493] Stack:
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.329451]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.350330] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.379651] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.409270] Call Trace:
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.411123]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.438935]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.459359]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.489327]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.510038]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.539568]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.560651]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.589561]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.610650]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.639767]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.659778]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.680331]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.709047]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.729603]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.749834]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.770162]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.789780]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.810167]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.830311]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.849984] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.918928] Call Trace:
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.920784]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.949118]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.969546]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31311.999521]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.020127]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.049400]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.070524]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.099787]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.120591]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.149766]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.169613]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.189840]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.210367]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.239429]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.259251]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.279822]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.299241]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.319634]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:50:35 atlas-tds-mds1 kernel: [31312.340024]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31394.446155] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31394.459630] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31394.730161] CPU 9 
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31394.731169] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.010201] 
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.011427] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.050447] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.071692] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.090990] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.120107] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.140863] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.161189] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.190338] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.210894] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.240900] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.260326] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.280865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.309967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.330581] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.360754] Stack:
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.362285]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.390211] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.411288] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.441169] Call Trace:
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.450395]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.470817]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.491443]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.521251]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.550468]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.571382]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.600877]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.621548]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.650748]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.671515]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.691382]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.711584]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.740570]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.761087]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.781112]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.801657]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.821118]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.841585]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.870240]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.881364] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.950393] Call Trace:
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.952216]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31395.980407]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.000804]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.030603]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.051556]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.080845]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.101934]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.131061]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.151795]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.181061]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.200891]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.221354]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.250507]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.270926]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.290933]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.311455]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.330733]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.351162]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:51:59 atlas-tds-mds1 kernel: [31396.371640]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31478.477005] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31478.491145] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31478.761060] CPU 9 
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31478.762290] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.032583] 
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.041709] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.081342] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.102644] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.121726] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.142440] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.171611] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.192183] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.212643] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.241679] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.263073] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.282489] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.311703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.332166] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.352781] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.382865] Stack:
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.391723]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.412571] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.441925] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.471310] Call Trace:
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.473219]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.501587]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.521911]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.551747]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.572618]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.601831]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.622940]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.651984]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.673071]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.702211]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.722126]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.742369]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.763048]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.791825]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.811848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.832401]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.851871]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.872284]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.892594]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.912230] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.973273] Call Trace:
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31479.982494]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.002980]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.042239]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.063053]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.092143]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.113089]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.142533]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.163287]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.192597]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.213427]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.241627]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.262014]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.282594]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.303222]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.323075]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.351655]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.362810]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.391708]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:53:23 atlas-tds-mds1 kernel: [31480.412320]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31562.507854] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31562.522239] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31562.792531] CPU 9 
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31562.793614] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.063446] 
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.072455] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.103849] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.133414] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.152681] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.173244] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.193842] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.222715] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.243516] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.263981] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.293903] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.313199] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.333812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.362767] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.383462] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.413615] Stack:
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.422598]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.443307] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.472601] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.494244] Call Trace:
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.503449]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.523752]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.544077]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.573964]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.603062]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.624195]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.653488]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.674313]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.703830]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.724495]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.752723]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.773134]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.793670]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.814102]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.834159]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.862961]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.874188]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.902971]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.923185]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31563.942754] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.003594] Call Trace:
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.013019]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.033419]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.053961]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.083591]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.104578]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.133869]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.154970]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.183826]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.204676]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.233998]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.253670]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.274023]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.302946]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.323574]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.343613]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.364136]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.383517]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.404087]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:54:47 atlas-tds-mds1 kernel: [31564.424637]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:56:03 atlas-tds-mds1 kernel: [31640.484511] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 22:56:03 atlas-tds-mds1 kernel: [31640.512116] Lustre: Skipped 19 previous similar messages
Mar 31 22:56:03 atlas-tds-mds1 kernel: [31640.522861] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 22:56:03 atlas-tds-mds1 kernel: [31640.572285] Lustre: Skipped 19 previous similar messages
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31646.538702] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31646.553535] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31646.824049] CPU 9 
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31646.825049] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.095014] 
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.104296] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.135353] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.165170] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.184211] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.204742] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.225359] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.254359] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.275219] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.303987] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.325490] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.344941] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.384370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.404428] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.425139] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.455256] Stack:
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.464267]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.485067] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.514277] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.535815] Call Trace:
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.545033]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.565421]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.594312]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.615673]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.644920]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.674404]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.695450]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.724441]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.745393]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.774428]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.794424]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.814833]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.835393]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.864240]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.884515]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.905026]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.924339]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.944675]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.965183]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31647.984515] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.045629] Call Trace:
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.055138]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.075353]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.095500]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.125470]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.146115]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.175612]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.204957]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.225525]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.254752]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.275684]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.295655]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.315883]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.344814]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.365177]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.385373]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.405653]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.425220]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.445538]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:56:11 atlas-tds-mds1 kernel: [31648.474497]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31730.569547] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31730.584865] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31730.846056] CPU 9 
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31730.854939] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.125482] 
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.126699] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.165855] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.195494] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.206384] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.235526] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.256042] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.276608] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.305680] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.326234] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.355918] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.375566] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.396442] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.425177] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.445775] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.475996] Stack:
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.477618]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.505567] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.526688] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.556610] Call Trace:
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.565882]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.586202]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.606598]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.636281]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.657057]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.686476]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.715684]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.736477]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.765925]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.786679]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.806660]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.835353]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.855953]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.876294]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.896479]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.916784]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.936340]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.956676]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.976840]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31731.996483] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.065601] Call Trace:
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.067486]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.095869]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.116040]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.145843]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.166639]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.196093]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.216978]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.246100]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.267001]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.296303]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.316223]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.336464]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.356996]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.385929]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.405984]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.426277]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.445973]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.466469]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:57:35 atlas-tds-mds1 kernel: [31732.486785]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31814.600392] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31814.616371] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31814.886285] CPU 9 
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31814.887404] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.157469] 
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.166561] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.197637] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.227364] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.246571] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.267184] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.287856] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.317031] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.337567] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.366672] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.388175] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.407573] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.436700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.457334] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.477902] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.508061] Stack:
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.516963]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.537771] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.566980] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.588526] Call Trace:
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.597703]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.626582]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.646967]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.676986]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.697559]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.718647]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.747922]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.777019]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.797975]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.818478]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.846830]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.867224]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.887780]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.908359]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.936658]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.957025]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.968311]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31815.996684]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.017097]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.036893] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.097915] Call Trace:
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.107178]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.127687]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.148063]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.178091]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.207144]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.228293]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.257590]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.278456]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.307585]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.328307]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.348246]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.376853]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.397653]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.418027]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.438220]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.467047]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.478584]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.507189]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 22:58:59 atlas-tds-mds1 kernel: [31816.527551]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31898.631235] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31898.647637] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31898.918173] CPU 9 
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31898.919173] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.198498] 
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.199768] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.238532] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.268093] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.279218] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.308107] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.328673] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.349368] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.378335] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.398873] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.428878] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.448155] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.468873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.497802] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.518564] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.548778] Stack:
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.557977]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.578582] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.608039] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.629404] Call Trace:
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.638797]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.658978]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.688038]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.709559]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.738445]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.759692]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.788866]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.809630]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.839071]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.859784]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.887921]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.908364]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.928897]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.949435]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.980130]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31899.998181]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.009427]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.038170]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.058582]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.078010] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.139149] Call Trace:
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.148602]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.168988]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.189154]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.219020]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.239835]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.269179]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.298503]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.319065]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.348555]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.369568]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.389251]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.409747]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.438523]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.459089]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.479143]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.499642]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.519110]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.539308]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:00:23 atlas-tds-mds1 kernel: [31900.568196]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31982.662075] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31982.678783] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31982.949106] CPU 9 
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31982.950219] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.220284] 
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.229372] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.260719] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.290151] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.309571] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.330120] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.359182] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.379620] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.400302] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.429260] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.450859] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.470402] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.499240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.520004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.540742] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.570768] Stack:
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.579683]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.600425] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.629733] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.659684] Call Trace:
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.661564]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.689387]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.709733]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.739613]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.760476]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.789798]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.810992]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.840067]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.861149]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.890261]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.909973]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.930407]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.959313]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.979889]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31983.999909]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.020280]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.039734]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.060096]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.080599]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.100053] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.160947] Call Trace:
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.170269]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.190759]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.219365]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.240992]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.270124]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.291093]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.320541]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.341197]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.370539]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.391206]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.419421]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.439855]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.460394]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.480900]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.500881]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.529872]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.549575]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.570113]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:01:47 atlas-tds-mds1 kernel: [31984.590633]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:02:06 atlas-tds-mds1 kernel: [32003.077887] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 23:02:06 atlas-tds-mds1 kernel: [32003.116980] LustreError: Skipped 4 previous similar messages
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32066.692915] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32066.710370] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32066.980599] CPU 9 
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32066.981618] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.251425] 
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.260482] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.291712] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.321312] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.340525] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.371871] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.390911] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.411438] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.440368] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.461114] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.490994] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.510400] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.531003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.551675] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.580634] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.602224] Stack:
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.611358]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.640484] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.661240] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.691027] Call Trace:
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.692756]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.720719]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.741293]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.770961]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.791720]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.821122]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.842171]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.871481]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.892422]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.921486]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.941495]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.962157]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32067.990809]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.011297]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.031402]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.051705]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.071192]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.091736]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.111885]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.131571] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.200694] Call Trace:
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.202598]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.230693]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.251094]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.272509]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.301657]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.322715]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.352015]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.372607]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.401866]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.431139]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.451123]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.471483]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.491975]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.512358]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.540794]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.561189]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.572428]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.601170]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:03:11 atlas-tds-mds1 kernel: [32068.621549]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32150.723753] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32150.741414] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.011454] CPU 9 
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.012496] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.282842] 
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.291968] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.323313] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.352977] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.372215] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.392929] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.421684] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.442457] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.463187] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.492139] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.522071] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.541657] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.562272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.582953] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.611900] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.633509] Stack:
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.642465]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.671727] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.692669] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.722345] Call Trace:
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.724245]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.751831]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.772222]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.793677]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.822956]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.852226]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.873313]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.902582]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.923479]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.952664]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.972757]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32151.992967]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.022085]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.042618]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.062837]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.083133]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.102670]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.123164]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.143432]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.162918] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.232114] Call Trace:
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.233993]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.262176]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.282690]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.312388]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.333335]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.362859]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.383851]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.412823]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.433670]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.462611]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.482890]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.503049]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.523637]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.552422]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.583216]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.602268]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.613414]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.642142]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:04:35 atlas-tds-mds1 kernel: [32152.662502]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32234.754589] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32234.772709] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.043225] CPU 9 
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.044321] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.314452] 
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.323622] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.363131] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.384402] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.403857] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.424549] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.453362] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.474050] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.502979] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.523701] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.553381] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.564558] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.593578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.614244] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.643232] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.664842] Stack:
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.674095]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.703058] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.724122] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.753641] Call Trace:
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.763086]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.783506]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.803736]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.833555]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.854419]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.883707]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.904630]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.933728]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.954625]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32235.983718]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.003919]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.023936]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.044746]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.073622]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.093819]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.114112]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.133753]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.154166]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.174318]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.193980] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.254870] Call Trace:
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.264183]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.284669]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.304992]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.334790]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.363911]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.384807]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.414477]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.435202]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.464441]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.485374]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.513520]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.534115]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.554633]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.583396]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.603465]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.623796]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.634914]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.663608]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:05:59 atlas-tds-mds1 kernel: [32236.683983]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:06:03 atlas-tds-mds1 kernel: [32240.713477] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 23:06:03 atlas-tds-mds1 kernel: [32240.736578] Lustre: Skipped 21 previous similar messages
Mar 31 23:06:03 atlas-tds-mds1 kernel: [32240.755691] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 23:06:03 atlas-tds-mds1 kernel: [32240.796924] Lustre: Skipped 19 previous similar messages
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32318.785424] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32318.804075] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.074544] CPU 9 
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.075634] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.345683] 
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.354823] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.385961] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.415528] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.434799] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.455368] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.484234] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.505049] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.525536] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.554505] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.576001] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.595595] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.624498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.645246] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.665830] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.696005] Stack:
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.705108]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.725750] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.755300] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.785121] Call Trace:
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.794385]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.814876]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.835258]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.864996]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.885878]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.925526]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.945913]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.974864]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32319.995794]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.016348]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.044720]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.065128]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.085504]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.106088]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.126106]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.154913]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.165961]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.194914]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.215166]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.234544] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.295314] Call Trace:
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.304977]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.325334]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.345695]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.375390]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.396102]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.425596]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.455220]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.475996]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.505240]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.525942]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.545974]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.574646]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.595264]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.615944]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.635961]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.664898]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.676011]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.704934]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:07:23 atlas-tds-mds1 kernel: [32320.725123]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32402.816258] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32402.835453] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.106157] CPU 9 
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.115032] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.385667] 
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.387047] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.425897] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.447146] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.466619] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.486964] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.515994] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.536665] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.565638] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.586284] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.616227] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.635599] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.656139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.676918] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.705687] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.727485] Stack:
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.736573]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.765610] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.786711] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.816456] Call Trace:
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.825749]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.846110]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.866470]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.896300]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.917030]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.946697]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.967699]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32403.996704]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.026148]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.046867]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.066865]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.087055]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.116021]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.136554]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.156690]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.177101]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.196582]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.217077]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.245834]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.257079] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.326296] Call Trace:
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.335903]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.356264]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.376659]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.407125]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.427762]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.456853]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.486315]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.507050]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.527981]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.556933]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.576751]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.597401]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.626428]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.646587]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.666731]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.687102]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.706824]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.727127]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:08:47 atlas-tds-mds1 kernel: [32404.747430]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32486.847090] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32486.866765] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.136896] CPU 9 
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.137914] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.416998] 
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.418295] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.457182] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.478617] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.497842] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.518391] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.547403] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.568001] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.596854] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.617635] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.647440] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.666910] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.687671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.708190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.737374] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.767325] Stack:
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.768941]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.796973] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.818099] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.847715] Call Trace:
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.856967]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.877227]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.897672]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.927438]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.948161]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.977646]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32487.998696]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.027897]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.048696]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.077692]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.097831]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.118204]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.147092]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.167494]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.187671]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.208070]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.227577]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.247903]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.268248]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.287923] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.348944] Call Trace:
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.358330]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.378804]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.407555]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.429013]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.457964]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.487614]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.508689]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.537875]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.558806]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.587880]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.607806]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.628388]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.648711]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.677505]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.697480]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.718048]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.737564]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.757716]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:10:11 atlas-tds-mds1 kernel: [32488.778267]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32570.877921] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32570.898061] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.168008] CPU 9 
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.169168] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.439038] 
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.448083] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.479539] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.509064] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.528292] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.548988] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.569486] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.598516] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.619117] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.648166] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.669535] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.689135] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.709787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.738805] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.759416] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.789598] Stack:
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.798583]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.819347] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.848406] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.870121] Call Trace:
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.879266]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.899549]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.928323]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.949778]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32571.978953]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.000303]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.029542]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.050012]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.079336]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.099993]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.128387]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.148753]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.169216]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.189654]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.209975]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.238611]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.249931]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.278666]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.299051]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.318448] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.379401] Call Trace:
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.388869]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.409398]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.429903]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.459607]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.480144]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.509818]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.549785]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.569591]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.590469]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.619306]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.639315]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.659678]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.688811]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.709318]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.729213]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.749687]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.769404]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.789401]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:11:35 atlas-tds-mds1 kernel: [32572.810035]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32654.908751] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32654.929187] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.199331] CPU 9 
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.200442] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.470347] 
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.479439] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.510744] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.540424] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.559606] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.580187] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.600720] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.629826] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.650404] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.670914] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.700982] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.720418] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.749542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.769969] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.790697] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.820839] Stack:
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.829875]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.850630] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.879711] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.901476] Call Trace:
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.910560]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.930872]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.959709]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32655.981343]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.010513]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.040033]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.060756]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.081795]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.110826]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.131596]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.159886]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.180062]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.200825]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.221261]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.249720]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.269912]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.281284]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.310143]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.330317]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.349746] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.410876] Call Trace:
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.420335]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.440760]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.461362]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.490788]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.511586]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.541393]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.570586]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.591443]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.620618]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.641314]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.661273]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.689863]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.710677]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.730992]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.751044]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.779938]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.791073]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.811490]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:12:59 atlas-tds-mds1 kernel: [32656.840087]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32738.939580] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32738.951933] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.230608] CPU 9 
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.231628] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.501685] 
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.510752] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.541954] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.571549] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.582519] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.611116] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.631863] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.660925] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.681523] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.702044] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.731929] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.751549] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.772112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.801290] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.821825] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.851844] Stack:
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.860815]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.881583] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.902574] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.932379] Call Trace:
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.941498]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.961832]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32739.992851]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.012384]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.041537]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.062559]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.091908]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.112590]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.141743]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.162598]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.190976]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.211169]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.231762]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.252399]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.272434]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.301249]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.312395]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.341240]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.361674]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.381041] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.442005] Call Trace:
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.451453]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.471940]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.492533]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.522055]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.542976]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.572107]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.601582]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.622328]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.651727]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.672444]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.692402]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.721157]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.741606]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.762103]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.782461]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.802615]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.822351]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.842788]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:14:23 atlas-tds-mds1 kernel: [32740.871238]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:14:36 atlas-tds-mds1 kernel: [32753.353346] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 23:14:36 atlas-tds-mds1 kernel: [32753.386497] LustreError: Skipped 4 previous similar messages
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32822.970406] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32822.982811] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.252667] CPU 9 
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.261642] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.532401] 
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.533622] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.572690] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.593908] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.613218] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.642220] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.662994] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.683504] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.712469] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.733057] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.762922] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.782469] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.803055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.823713] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.852546] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.882786] Stack:
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.884334]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.912524] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.933599] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.963390] Call Trace:
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.972435]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32823.992676]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.012981]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.042803]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.063756]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.093055]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.114060]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.143213]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.164057]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.193062]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.213230]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.233424]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.262569]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.283055]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.302917]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.323529]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.342806]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.362909]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.383321]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.402917] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.464003] Call Trace:
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.473331]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.493694]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.522426]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.543926]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.572977]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.593937]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.623473]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.644046]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.673215]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.693999]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.713969]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.742683]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.763336]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.783746]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.803782]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.832520]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.843801]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.872544]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:15:47 atlas-tds-mds1 kernel: [32824.892947]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:16:39 atlas-tds-mds1 kernel: [32876.913378] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) reconnecting
Mar 31 23:16:39 atlas-tds-mds1 kernel: [32876.942500] Lustre: Skipped 20 previous similar messages
Mar 31 23:16:39 atlas-tds-mds1 kernel: [32876.961962] Lustre: atlastds-MDT0000: Client 98846250-6b04-9191-3660-3a16c1e56d27 (at 13@gni2) refused reconnection, still busy with 2 active RPCs
Mar 31 23:16:39 atlas-tds-mds1 kernel: [32877.002737] Lustre: Skipped 20 previous similar messages
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.001232] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.013520] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.284382] CPU 9 
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.293257] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.563846] 
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.565072] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.604127] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.633751] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.644654] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.673724] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.694472] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.723360] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.743989] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.764519] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.794567] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.814013] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.834614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.863613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.884173] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.914240] Stack:
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.915776]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.944055] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.965210] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32907.994801] Call Trace:
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.003928]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.024507]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.044896]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.074814]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.104012]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.125062]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.154294]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.175074]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.204574]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.225048]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.253589]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.274006]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.294402]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.314885]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.335048]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.363855]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.375091]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.403868]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.424085]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.435124] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.504644] Call Trace:
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.514084]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.534419]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.554660]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.584495]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.605575]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.634877]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.664356]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.685103]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.714384]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.735254]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.755231]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.783777]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.804373]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.824911]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.844772]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.865243]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.884649]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.905118]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:17:11 atlas-tds-mds1 kernel: [32908.933866]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.032057] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.044405] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.315063] CPU 9 
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.316135] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.585707] 
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.594757] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.626237] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.655610] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.674887] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.695627] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.724692] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.745663] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.766164] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.795118] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.816622] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.836016] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.864919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.885710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.906217] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.936468] Stack:
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.945497]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.965995] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32991.995195] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.016769] Call Trace:
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.026095]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.054793]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.075181]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.096649]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.125829]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.155275]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.176725]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.205413]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.226237]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.255408]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.275272]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.295835]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.316303]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.345298]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.365477]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.385692]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.405293]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.425712]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.446028]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.476164] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.535919] Call Trace:
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.545379]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.565929]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.586118]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.615968]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.636811]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.666159]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.695638]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.716617]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.745708]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.766708]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.786475]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.815324]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.835936]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.856360]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.876592]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.905270]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.916385]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.945345]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:18:35 atlas-tds-mds1 kernel: [32992.965694]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.062880] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.075705] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.345787] CPU 9 
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.346970] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.617062] 
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.625926] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.657214] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.686713] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.705930] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.726565] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.747229] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.776187] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.796738] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.826002] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.847727] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.867163] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.896176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.916796] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.937614] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.967988] Stack:
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.976855]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33075.997766] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.027061] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.056653] Call Trace:
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.058527]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.086311]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.106716]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.136547]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.157287]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.186721]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.207556]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.236735]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.257486]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.286578]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.306354]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.327032]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.347375]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.376229]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.396391]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.416814]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.436438]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.456680]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.477059]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.496476] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.557413] Call Trace:
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.566875]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.587313]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.607581]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.637541]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.666723]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.687638]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.717091]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.737819]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.767130]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.788001]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.816384]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.836664]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.857401]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.877916]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.897879]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.926590]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.937686]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.966593]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:19:59 atlas-tds-mds1 kernel: [33076.986891]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.093702] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.107012] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.377536] CPU 9 
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.378637] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.648872] 
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.657984] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.697901] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.719076] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.738206] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.767301] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.787835] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.808415] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.837463] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.858224] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.887874] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.907506] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.938766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.957733] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33159.978436] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.008490] Stack:
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.017540]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.038323] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.067556] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.088991] Call Trace:
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.098190]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.118669]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.147414]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.168909]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.197999]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.219079]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.248434]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.269413]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.298314]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.319138]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.339036]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.367736]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.388399]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.408988]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.428842]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.457753]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.468848]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.497717]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.518139]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.537762] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.598685] Call Trace:
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.607980]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.628375]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.648719]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.678703]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.699308]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.728822]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.758294]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.778870]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.808269]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.828994]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.848956]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.869175]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.898072]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.918657]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.938789]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.959316]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.978683]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33160.999160]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:21:23 atlas-tds-mds1 kernel: [33161.027913]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.124540] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.138359] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.399691] CPU 9 
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.408796] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.678918] 
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.680141] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.719336] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.749109] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.759878] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.788959] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.809606] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.830202] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.859208] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.879567] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.909568] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.928979] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.949808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.970196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33243.999234] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.029443] Stack:
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.031129]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.059044] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.079925] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.109638] Call Trace:
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.118908]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.139381]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.159656]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.189754]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.210459]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.239713]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.269195]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.290003]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.319341]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.339904]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.359878]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.380217]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.409071]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.429771]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.449765]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.470196]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.489906]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.510267]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.530314]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.549942] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.619038] Call Trace:
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.620926]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.649090]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.669468]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.690736]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.720061]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.749520]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.770431]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.799559]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.820631]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.849557]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.869555]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.890256]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.918992]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.939588]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.959578]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.980180]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33244.999548]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33245.019853]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:22:47 atlas-tds-mds1 kernel: [33245.040300]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.155418] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.169646] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.440339] CPU 9 
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.441480] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.710782] 
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.719833] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.750984] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.780679] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.799876] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.820433] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.841137] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.870102] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.890719] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.911451] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.941291] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.960718] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33327.981237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.010385] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.031071] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.061299] Stack:
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.070365]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.091078] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.120188] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.141807] Call Trace:
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.150975]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.180057]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.200476]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.221731]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.251048]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.280608]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.301732]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.331054]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.351926]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.380889]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.400857]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.421121]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.450371]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.470703]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.490898]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.511099]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.530569]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.551127]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.571575]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.591105] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.660210] Call Trace:
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.662064]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.690451]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.711015]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.740778]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.761589]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.791103]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.812082]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.841073]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.861959]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.891280]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.911252]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.931676]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.960564]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33328.980902]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33329.000910]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33329.021281]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33329.040757]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33329.061113]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:24:11 atlas-tds-mds1 kernel: [33329.081463]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:24:36 atlas-tds-mds1 kernel: [33353.590981] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 23:24:36 atlas-tds-mds1 kernel: [33353.629909] LustreError: Skipped 3 previous similar messages
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.186294] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.200992] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.471435] CPU 9 
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.472434] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.742296] 
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.751322] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.782631] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.812116] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.831192] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.851917] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.872646] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.901450] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.922224] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.951195] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.972710] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33411.992320] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.021259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.041980] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.062718] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.092732] Stack:
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.101829]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.122570] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.151444] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.173096] Call Trace:
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.182253]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.202586]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.231359]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.252823]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.281937]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.302930]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.332521]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.353164]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.382215]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.402956]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.431405]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.462077]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.481999]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.502345]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.522450]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.542852]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.562556]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.582931]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.611618]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.622853] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.692043] Call Trace:
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.701654]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.722094]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.742470]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.772278]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.793006]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.822387]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.851913]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.872711]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.893376]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.922548]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.942491]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.962984]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33412.991825]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.012342]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.032560]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.052724]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.072398]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.092612]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:25:35 atlas-tds-mds1 kernel: [33413.113053]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.217169] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.232394] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.502524] CPU 9 
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.503530] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.773367] 
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.782414] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.813780] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.843379] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.862708] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.883461] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.904140] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.933064] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.953798] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33495.982692] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.004059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.023732] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.052566] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.073151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.093962] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.124058] Stack:
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.132976]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.153926] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.182939] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.204650] Call Trace:
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.213877]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.234045]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.262839]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.284289]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.313414]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.334311]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.363986]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.384484]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.413837]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.434637]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.462906]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.483096]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.503863]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.532759]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.552929]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.573162]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.592799]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.613193]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.633398]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.653028] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.713778] Call Trace:
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.723249]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.743582]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.763769]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.793793]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.814737]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.843951]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.873444]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.894339]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.923456]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.944380]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.964459]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33496.993104]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.013664]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.034243]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.054208]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.083138]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.094256]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.123121]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:26:59 atlas-tds-mds1 kernel: [33497.143418]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:27:33 atlas-tds-mds1 kernel: [33530.800047] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 23:27:33 atlas-tds-mds1 kernel: [33530.826058] Lustre: Skipped 20 previous similar messages
Mar 31 23:27:33 atlas-tds-mds1 kernel: [33530.836813] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 23:27:33 atlas-tds-mds1 kernel: [33530.886484] Lustre: Skipped 20 previous similar messages
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.248043] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.263566] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.534584] CPU 9 
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.543275] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.814161] 
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.825860] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.854798] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.884675] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.903852] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.924403] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.945161] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.973993] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33579.994587] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.016033] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.045156] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.064609] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.085144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.114334] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.134855] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.164881] Stack:
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.174075]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.194826] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.223906] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.245401] Call Trace:
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.254672]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.274894]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.295368]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.325347]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.354420]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.375502]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.404938]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.425717]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.454776]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.475480]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.495274]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.524078]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.544627]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.565201]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.585324]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.614069]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.625227]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.654092]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.674619]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.694060] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.755278] Call Trace:
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.764729]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.785265]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.805589]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.835613]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.864566]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.885574]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.914988]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.935735]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.965019]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33580.985727]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.005675]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.034617]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.055196]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.075671]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.095691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.124265]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.135444]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.155744]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:28:23 atlas-tds-mds1 kernel: [33581.184485]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.278915] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.294848] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.565422] CPU 9 
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.566517] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.836162] 
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.845260] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.876530] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.906087] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.925158] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.945722] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.966397] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33663.995132] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.015683] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.036268] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.066148] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.085558] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.106320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.126769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.156088] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.185758] Stack:
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.187386]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.215620] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.236684] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.266222] Call Trace:
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.275651]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.296021]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.316377]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.346231]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.366928]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.396449]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.425818]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.446606]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.475680]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.496708]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.516585]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.545363]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.565914]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.586359]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.606681]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.626776]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.646621]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.666790]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.695651]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.706966] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.775987] Call Trace:
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.785654]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.805832]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.826351]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.856021]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.876763]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.906072]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.927033]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.956258]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33664.977136]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.016743]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.035885]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.056114]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.076922]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.097069]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.125563]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.145926]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.157017]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.185962]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:29:47 atlas-tds-mds1 kernel: [33665.206152]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.309785] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.326331] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.596697] CPU 9 
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.597708] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.867959] 
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.876947] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.916560] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.937892] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.957140] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33747.977884] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.006872] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.027600] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.056461] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.077051] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.106872] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.117914] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.147108] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.167745] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.196691] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.218456] Stack:
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.227418]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.256518] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.277619] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.307535] Call Trace:
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.316562]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.336946]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.357511]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.387357]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.408311]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.437638]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.466981]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.487775]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.517039]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.537829]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.557606]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.578064]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.606918]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.627389]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.647625]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.667982]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.687450]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.707990]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.728172]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.747782] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.816902] Call Trace:
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.818781]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.846968]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.867343]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.897348]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.917917]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.947376]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.968603]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33748.997612]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.018361]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.047621]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.067703]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.087763]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.108322]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.137226]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.157282]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.177626]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.197304]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.217669]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:31:11 atlas-tds-mds1 kernel: [33749.237990]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.340655] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.357663] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.627792] CPU 9 
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.628834] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.907666] 
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.908984] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.948059] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.969329] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33831.988613] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.009243] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.038304] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.058862] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.087902] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.108582] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.138546] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.157842] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.178694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.199062] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.228007] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.258231] Stack:
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.259847]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.287963] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.308944] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.338766] Call Trace:
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.348033]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.368438]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.388793]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.418659]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.439374]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.479352]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.499120]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.519838]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.549256]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.569833]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.598147]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.618585]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.639378]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.668006]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.688189]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.708559]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.728044]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.748588]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.768923]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.788424] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.849571] Call Trace:
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.858835]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.879025]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.899467]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.929229]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.949864]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33832.979076]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.008499]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.029176]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.058496]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.079261]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.099226]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.119641]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.148332]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.168961]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.188987]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.209367]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.228607]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.249102]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:32:35 atlas-tds-mds1 kernel: [33833.269664]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.371523] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.388772] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.658878] CPU 9 
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.659986] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.930206] 
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.939195] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33915.970406] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.000119] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.019236] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.039795] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.060551] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.089603] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.110366] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.139408] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.160996] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.180322] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.209095] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.229513] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.250458] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.280399] Stack:
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.289328]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.310247] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.339280] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.360817] Call Trace:
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.370111]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.390432]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.410956]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.440630]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.469840]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.490750]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.520179]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.540966]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.570028]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.590957]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.619256]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.639677]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.660493]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.689275]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.709285]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.729676]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.740731]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.769508]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.789719]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.809284] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.870297] Call Trace:
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.879735]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.900277]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.920515]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.950562]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33916.979832]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.000901]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.030174]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.050897]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.080247]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.100904]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.120917]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.149618]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.170227]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.190928]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.210906]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.239654]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.250905]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.279729]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:33:59 atlas-tds-mds1 kernel: [33917.299978]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.402389] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.420114] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.690255] CPU 9 
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.691374] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.962161] 
Mar 31 23:35:23 atlas-tds-mds1 kernel: [33999.970932] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.010538] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.031632] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.050969] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.071449] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.100541] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.121368] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.150361] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.170961] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.201132] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.220540] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.241131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.261820] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.290605] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.320796] Stack:
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.322338]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.350481] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.371576] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.401405] Call Trace:
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.410667]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.430894]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.451461]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.481230]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.502020]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.531458]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.560920]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.581682]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.610958]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.631729]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.651722]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.672010]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.700936]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.721594]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.741583]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.761981]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.781430]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.801782]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.822040]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.841974] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.911031] Call Trace:
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.920683]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.941040]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.961284]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34000.991295]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.011917]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.041268]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.062573]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.091475]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.112658]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.141483]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.161696]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.181856]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.210955]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.231542]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.251361]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.271875]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.291367]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.311772]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:35:23 atlas-tds-mds1 kernel: [34001.332067]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34083.433255] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34083.451642] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34083.722064] CPU 9 
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34083.723136] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.001808] 
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.003031] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.042214] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.063416] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.082885] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.111845] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.132575] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.152927] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.182056] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.202673] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.232447] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.251913] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.272684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.293172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.322120] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.352136] Stack:
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.353675]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.381896] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.403042] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.432698] Call Trace:
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.441983]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.462360]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.482817]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.512453]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.533143]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.562774]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.592221]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.612976]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.642421]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.663036]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.682965]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.703273]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.732292]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.752660]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.772760]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.793310]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.812798]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.833065]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.861945]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.873310] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.942341] Call Trace:
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.951970]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.972327]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34084.992737]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.022576]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.043335]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.072781]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.104304]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.123148]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.143997]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.173228]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.193070]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.213284]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.242256]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.262663]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.282554]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.303202]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.322691]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.343108]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:36:47 atlas-tds-mds1 kernel: [34085.363538]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:37:06 atlas-tds-mds1 kernel: [34103.872745] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.67@o2ib, operation ost_connect failed with -30.
Mar 31 23:37:06 atlas-tds-mds1 kernel: [34103.909571] LustreError: Skipped 4 previous similar messages
Mar 31 23:37:33 atlas-tds-mds1 kernel: [34131.025204] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) reconnecting
Mar 31 23:37:33 atlas-tds-mds1 kernel: [34131.049733] Lustre: Skipped 19 previous similar messages
Mar 31 23:37:33 atlas-tds-mds1 kernel: [34131.060608] Lustre: atlastds-MDT0000: Client 2892c2bc-6435-ceee-96aa-67a7e6401f59 (at 12@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 23:37:33 atlas-tds-mds1 kernel: [34131.109895] Lustre: Skipped 19 previous similar messages
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34167.464119] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34167.482956] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34167.753184] CPU 9 
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34167.754331] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.023999] 
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.033103] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.064554] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.094053] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.113100] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.133713] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.154515] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.183348] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.204014] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.233048] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.254770] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.273970] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.294637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.323586] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.344449] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.374379] Stack:
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.383416]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.404159] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.433431] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.455032] Call Trace:
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.464292]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.484545]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.513187]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.534892]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.563932]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.584973]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.614268]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.635108]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.664306]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.684975]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.713373]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.733942]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.754302]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.783184]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.803407]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.823564]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.834653]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.863642]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.883986]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.903629] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.964728] Call Trace:
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.974022]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34168.994580]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.014795]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.044720]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.073793]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.094700]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.124088]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.145082]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.174323]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.195227]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.223383]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.243757]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.264482]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.284918]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.313422]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.333582]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.344912]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.373605]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:38:11 atlas-tds-mds1 kernel: [34169.394005]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34251.494982] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34251.514221] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34251.784116] CPU 9 
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34251.785147] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.055285] 
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.064389] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.095653] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.125174] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.144464] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.165206] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.185786] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.214895] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.235453] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.264524] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.286030] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.305620] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.334565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.355268] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.384366] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.406171] Stack:
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.425769]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.444939] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.465816] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.495430] Call Trace:
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.505013]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.525210]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.545561]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.575552]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.596265]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.625597]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.655229]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.675963]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.705262]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.725967]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.745772]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.774526]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.795292]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.815783]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.836009]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.864772]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.876016]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.904771]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.925213]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34252.944799] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.005400] Call Trace:
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.014797]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.035015]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.055374]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.085392]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.106124]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.135434]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.156455]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.185644]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.206552]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.235690]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.255626]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.275716]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.304692]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.325268]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.345142]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.365490]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.384958]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.405510]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:39:35 atlas-tds-mds1 kernel: [34253.425701]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34335.525844] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34335.545524] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34335.815974] CPU 9 
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34335.817041] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.086805] 
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.095917] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.127183] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.156879] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.175946] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.196699] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.225592] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.246202] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.266940] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.296016] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.317536] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.337148] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.366096] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.386395] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.407102] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.437336] Stack:
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.446411]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.467120] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.496293] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.525662] Call Trace:
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.527448]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.555715]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.576284]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.606125]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.626752]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.656150]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.677280]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.706367]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.727161]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.756227]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.776166]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.796458]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.817095]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.845848]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.865825]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.886227]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.897333]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.926218]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.946474]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34336.965906] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.027043] Call Trace:
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.036447]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.056884]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.077224]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.107085]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.127762]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.157316]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.186735]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.207604]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.236766]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.257641]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.285983]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.306408]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.327024]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.347529]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.367621]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.396442]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.407527]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.436447]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:40:59 atlas-tds-mds1 kernel: [34337.456827]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34419.556705] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34419.576829] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34419.847678] CPU 9 
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34419.856411] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.127179] 
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.128408] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.167608] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.188890] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.207969] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.236892] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.257300] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.277995] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.306929] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.327472] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.357610] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.376993] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.397405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.418062] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.446963] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.468716] Stack:
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.477588]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.498404] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.527587] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.557007] Call Trace:
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.558918]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.587090]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.607618]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.628949]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.658077]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.687468]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.708549]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.737487]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.758401]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.787658]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.807471]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.828054]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.848615]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.877401]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.897549]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.917769]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.937352]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.957764]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.978250]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34420.997561] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.058673] Call Trace:
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.067954]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.088529]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.117205]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.138566]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.167616]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.188772]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.218413]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.239120]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.268399]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.289057]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.317286]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.337885]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.358343]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.378722]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.398866]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.427547]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.438698]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.467407]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:42:23 atlas-tds-mds1 kernel: [34421.487734]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34503.587565] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34503.608157] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34503.878464] CPU 9 
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34503.879464] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.158123] 
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.159383] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.198519] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.219686] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.239094] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.259637] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.288801] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.309505] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.338455] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.358864] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.388891] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.408295] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.428855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.449540] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.478296] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.499900] Stack:
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.509107]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.529898] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.559076] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.588854] Call Trace:
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.590587]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.618357]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.638753]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.660190]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.689341]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.710200]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.739562]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.768786]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.789697]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.818798]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.838796]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.859048]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.879720]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.899942]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.919938]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.948642]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.959915]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34504.988700]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.009263]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.038999] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.099504] Call Trace:
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.108983]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.129581]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.149886]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.179486]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.200245]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.229540]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.258926]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.279756]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.309013]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.329957]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.349601]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.370128]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.399278]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.419939]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.440067]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.460217]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.479841]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.500280]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:43:47 atlas-tds-mds1 kernel: [34505.528909]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34587.618423] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34587.630662] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34587.900483] CPU 9 
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34587.909345] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.179861] 
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.181129] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.220064] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.241289] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.260453] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.280980] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.310064] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.330660] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.359718] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.380132] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.410107] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.429576] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.450284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.470950] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.499973] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.529979] Stack:
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.531521]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.559812] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.580563] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.610401] Call Trace:
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.619630]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.640042]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.660295]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.690253]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.710848]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.740286]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.761376]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.790486]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.811555]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.840691]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.860522]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.881159]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.909984]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.930637]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.950531]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.970913]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34588.990336]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.010785]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.031019]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.050574] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.111614] Call Trace:
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.120803]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.141157]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.169859]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.191313]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.220475]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.241536]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.271033]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.291717]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.320894]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.341555]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.369933]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.390172]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.410897]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.431374]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.451449]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.480361]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.499960]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.520213]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:45:11 atlas-tds-mds1 kernel: [34589.540712]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34671.649280] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34671.661510] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34671.931291] CPU 9 
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34671.932439] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.202352] 
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.211497] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.251184] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.272462] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.291875] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.312444] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.341425] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.361920] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.390870] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.411584] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.441264] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.452285] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.491926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.511447] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.532045] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.562076] Stack:
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.571311]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.592020] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.621122] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.642731] Call Trace:
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.652136]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.672402]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.700993]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.722633]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.751753]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.772851]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.802322]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.831655]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.852590]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.881774]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.901839]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.922014]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.942533]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.971214]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34672.991305]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.011803]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.031282]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.051526]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.072086]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.091493] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.152449] Call Trace:
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.161699]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.182298]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.202588]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.232694]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.261763]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.282896]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.312157]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.332918]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.361983]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.382738]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.402499]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.431260]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.452006]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.472347]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.492523]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.521458]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.532593]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.561282]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:46:35 atlas-tds-mds1 kernel: [34673.581723]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34755.680137] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34755.692477] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34755.962379] CPU 9 
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34755.963428] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.233679] 
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.242866] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.282714] RIP: 0010:[<ffffffff8150eefc>]  [<ffffffff8150eefc>] _spin_lock+0x1c/0x30
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.303909] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.323223] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.352159] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.372618] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.393289] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.422178] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.442784] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.472618] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.492215] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.512746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.533389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.562426] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.584050] Stack:
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.593327]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.622214] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.642993] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.672816] Call Trace:
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.674656]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.702476]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.722692]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.752682]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.773539]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.802742]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.823784]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.853074]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.873999]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.903302]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.923128]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.943501]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.972393]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34756.992787]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.012789]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.033192]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.052588]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.072977]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.093523]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.113207] Code: 81 2f 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.174167] Call Trace:
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.183390]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.203737]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.232493]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.253802]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.283088]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.304291]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.333670]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.354324]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.383735]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.404281]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.432567]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.452928]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.473747]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.502599]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.522651]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.543041]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.554022]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.583058]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:47:59 atlas-tds-mds1 kernel: [34757.603347]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:48:33 atlas-tds-mds1 kernel: [34791.627129] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 23:48:33 atlas-tds-mds1 kernel: [34791.655897] Lustre: Skipped 20 previous similar messages
Mar 31 23:48:33 atlas-tds-mds1 kernel: [34791.658585] Lustre: atlastds-MDT0000: Client e26fea1c-02cb-fa50-94b2-cc7983b9a3a1 (at 3@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 23:48:33 atlas-tds-mds1 kernel: [34791.658587] Lustre: Skipped 20 previous similar messages
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34839.710992] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34839.723399] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34839.993858] CPU 9 
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34839.994863] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.273674] 
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.275009] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.314001] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.335414] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.354551] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.383447] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.404221] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.424857] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.453877] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.474422] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.504585] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.524059] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.544679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.573543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.594307] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.624296] Stack:
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.625837]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.653936] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.674811] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.704714] Call Trace:
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.713986]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.734356]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.754739]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.784803]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.805472]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.834764]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.863993]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.884780]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.914213]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.934981]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.954782]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34840.975247]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.004078]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.024591]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.044688]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.065109]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.084702]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.105098]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.125491]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.144941] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.213986] Call Trace:
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.215834]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.244041]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.264606]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.294420]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.314984]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.344383]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.365317]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.394414]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.415347]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.444409]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.464279]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.484801]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.505166]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.533905]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.554085]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.574608]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.593905]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.614356]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:49:23 atlas-tds-mds1 kernel: [34841.634673]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:49:36 atlas-tds-mds1 kernel: [34854.124744] LustreError: 11-0: atlastds-OST00f3-osc-MDT0000: Communicating with 10.36.226.71@o2ib, operation ost_connect failed with -19.
Mar 31 23:49:36 atlas-tds-mds1 kernel: [34854.159017] LustreError: Skipped 4 previous similar messages
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34923.741846] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34923.754535] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.024804] CPU 9 
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.025897] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.295689] 
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.304687] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.335832] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.365674] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.384934] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.405511] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.426247] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.455350] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.475910] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.504986] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.526424] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.545904] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.574919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.595563] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.616095] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.646161] Stack:
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.655231]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.676072] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.705334] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.726815] Call Trace:
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.736012]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.756337]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.785071]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.806718]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.835974]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.856920]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.886243]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.925832]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.946308]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.967032]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34924.995236]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.015464]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.036265]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.065019]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.085037]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.105800]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.125017]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.145422]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.166007]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.185389] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.246400] Call Trace:
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.255831]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.276023]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.296558]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.326487]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.355691]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.376790]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.405964]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.426652]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.455950]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.476655]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.505170]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.525533]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.546146]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.566450]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.586629]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.615585]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.626649]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.655435]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:50:47 atlas-tds-mds1 kernel: [34925.675966]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35007.772700] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35007.786003] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.056275] CPU 9 
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.057282] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.327122] 
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.336246] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.367676] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.397160] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.416240] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.436996] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.457494] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.486640] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.507199] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.536174] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.557661] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.577076] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.606111] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.626867] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.647417] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.677914] Stack:
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.687033]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.707562] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.736738] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.766179] Call Trace:
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.768193]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.796235]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.816657]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.838230]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.867390]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.896812]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.917930]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.946998]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.967984]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35008.997020]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.016862]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.037514]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.057895]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.086703]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.106904]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.127283]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.147107]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.167983]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.187828]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.207290] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.268214] Call Trace:
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.277516]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.297749]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.326592]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.347939]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.377176]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.398222]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.427582]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.448436]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.477618]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.506825]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.526666]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.547080]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.567679]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.588022]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.608061]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.636906]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.647960]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.676917]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:52:11 atlas-tds-mds1 kernel: [35009.697388]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35091.803552] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35091.817144] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.078672] CPU 9 
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.087608] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.358317] 
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.377791] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.408674] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.438142] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.457385] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.477984] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.498561] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.527610] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.548276] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.568861] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.598920] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.618201] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.638740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.667856] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.688460] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.718439] Stack:
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.727445]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.748252] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.777460] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.798989] Call Trace:
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.808132]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.828623]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.848979]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.878673]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.899595]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.928890]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.958372]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35092.979112]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.008451]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.028933]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.049063]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.077644]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.098241]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.118757]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.138785]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.159167]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.178905]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.199272]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.228053]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.239114] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.308168] Call Trace:
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.317749]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.338359]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.358645]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.388321]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.409077]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.438536]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.459586]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.488554]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.509432]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.538632]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.558655]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.578628]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.599267]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.628350]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.648275]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.668730]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.688126]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.708468]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:53:35 atlas-tds-mds1 kernel: [35093.728788]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35175.834403] BUG: soft lockup - CPU#9 stuck for 68s! [ldlm_elt:11787]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35175.848626] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.118953] CPU 9 
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.120111] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.399036] 
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.400298] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.439269] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.460484] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.479769] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.500185] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.529287] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.550078] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.578913] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.599497] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.629526] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.649125] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.669714] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.690379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.719159] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.749360] Stack:
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.750986]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.778965] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.800093] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.829775] Call Trace:
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.839021]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.859453]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.879829]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.909667]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.930224]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.959686]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35176.980639]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.009707]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.030581]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.059859]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.080025]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.100226]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.129184]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.149667]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.169729]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.190170]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.209541]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.230091]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.250257]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.269752] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.331003] Call Trace:
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.340173]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.360491]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.389067]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.410541]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.439494]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.460728]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.490052]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.510746]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.539905]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.571068]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.589307]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.609543]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.630242]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.659107]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.679416]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.699519]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.719171]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.739579]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:54:59 atlas-tds-mds1 kernel: [35177.759916]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35259.865254] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35259.879774] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.149859] CPU 9 
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.150970] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.420940] 
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.430186] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.461403] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.491116] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.510428] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.530960] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.551513] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.580624] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.601297] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.630533] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.651898] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.671485] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.700713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.721254] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.750095] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.771915] Stack:
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.780903]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.801696] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.831027] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.860891] Call Trace:
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.870141]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.890360]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.910918]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.940810]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.961492]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35260.990951]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.011830]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.040960]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.061796]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.090855]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.110898]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.131179]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.151822]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.180609]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.200856]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.221247]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.240484]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.260879]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.281333]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.300884] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.361951] Call Trace:
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.371253]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.391812]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.420344]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.442035]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.470920]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.492228]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.521524]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.542416]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.571577]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.592269]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.620414]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.641026]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.661617]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.681994]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.702208]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.730623]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.742028]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.770537]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:56:23 atlas-tds-mds1 kernel: [35261.790943]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35343.896115] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35343.911060] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.172766] CPU 9 
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.181724] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.452539] 
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.461443] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.492785] RIP: 0010:[<ffffffff8150eefe>]  [<ffffffff8150eefe>] _spin_lock+0x1e/0x30
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.522413] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.541865] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.562423] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.591504] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.612098] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.632768] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.661744] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.683020] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.702745] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.731433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.752021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.772518] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.802832] Stack:
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.812013]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.832662] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.861814] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.883228] Call Trace:
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.892415]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.912739]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.941692]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.963158]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35344.992280]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.032395]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.052985]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.073497]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.102832]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.123332]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.151793]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.172163]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.192839]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.221621]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.241752]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.262197]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.281582]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.302008]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.322358]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.341766] Code: 00 00 00 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.402978] Call Trace:
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.412400]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.432897]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.461727]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.483386]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.512473]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.533552]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.563025]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.592258]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.613206]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.642260]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.662313]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.682654]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.703227]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.731961]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.752227]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.772859]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.792164]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.812569]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:57:47 atlas-tds-mds1 kernel: [35345.832901]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:58:33 atlas-tds-mds1 kernel: [35391.851969] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) reconnecting
Mar 31 23:58:33 atlas-tds-mds1 kernel: [35391.879531] Lustre: Skipped 19 previous similar messages
Mar 31 23:58:33 atlas-tds-mds1 kernel: [35391.890409] Lustre: atlastds-MDT0000: Client e3c29ba9-9a7b-df10-db91-9d83da374912 (at 29@gni2) refused reconnection, still busy with 1 active RPCs
Mar 31 23:58:33 atlas-tds-mds1 kernel: [35391.939990] Lustre: Skipped 21 previous similar messages
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35427.926976] BUG: soft lockup - CPU#9 stuck for 67s! [ldlm_elt:11787]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35427.942548] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.204029] CPU 9 
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.212868] Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mbcache lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu autofs4 ipmi_devintf 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sd_mod crc_t10dif wmi acpi_pad shpchp megaraid_sas microcode dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support mlx4_en ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc mlx4_core igb dca ptp pps_core [last unloaded: scsi_wait_scan]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.483514] 
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.492566] Pid: 11787, comm: ldlm_elt Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Dell Inc. PowerEdge C6220 II/09N44V
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.523870] RIP: 0010:[<ffffffff8150ef01>]  [<ffffffff8150ef01>] _spin_lock+0x21/0x30
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.553391] RSP: 0018:ffff880802a91c30  EFLAGS: 00000297
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.572691] RAX: 0000000000000033 RBX: ffff880802a91c30 RCX: ffff880802a91da0
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.593276] RDX: 0000000000000032 RSI: 0000000000000000 RDI: ffff88075db65bd8
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.614032] RBP: ffffffff8100bb8e R08: 000000000000000e R09: 0000000000000002
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.642826] R10: 0000000000000402 R11: 0000000200000400 R12: ffff88075dcc72b8
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.663590] R13: ffffffffa03f2da4 R14: ffff880802a91bc0 R15: ffffffffa06e59d2
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.684078] FS:  00007f417c5b7700(0000) GS:ffff88089c420000(0000) knlGS:0000000000000000
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.714016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.733433] CR2: 00007f9c44316000 CR3: 0000000001a85000 CR4: 00000000001407e0
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.754254] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.783099] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.803647] Process ldlm_elt (pid: 11787, threadinfo ffff880802a90000, task ffff880802a88040)
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.833671] Stack:
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.842701]  ffff880802a91c50 ffffffffa06c0070 ffff88075dcc7240 ffff88075dcf7d80
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.863644] <d> ffff880802a91c70 ffffffffa06c432d ffff88075dcf7d80 ffff880802a91da0
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.892716] <d> ffff880802a91cb0 ffffffffa06c67b0 ffff880802a91ca0 ffff88075e862d80
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.914443] Call Trace:
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.923937]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.943913]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.972823]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35428.994123]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.023416]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.044320]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.073780]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.094502]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.123876]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.144591]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.164395]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.193273]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.213650]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.234219]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.254337]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.283098]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.294206]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.322951]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.343271]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.354541] Code: 01 74 05 e8 b2 2d d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 <eb> f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.423983] Call Trace:
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.433350]  [<ffffffffa06c0070>] ? lock_res_and_lock+0x30/0x50 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.453699]  [<ffffffffa06c432d>] ? ldlm_lock_cancel+0x2d/0x1e0 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.474000]  [<ffffffffa06c67b0>] ? ldlm_cancel_locks_for_export_cb+0x90/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.504101]  [<ffffffffa03f45cf>] ? cfs_hash_for_each_relax+0x17f/0x360 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.524808]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.554131]  [<ffffffffa06c6720>] ? ldlm_cancel_locks_for_export_cb+0x0/0x200 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.583549]  [<ffffffffa03f5e1e>] ? cfs_hash_for_each_empty+0xfe/0x1e0 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.604175]  [<ffffffffa06c10af>] ? ldlm_cancel_locks_for_export+0x2f/0x40 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.633445]  [<ffffffffa06d0454>] ? server_disconnect_export+0x64/0x1a0 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.654354]  [<ffffffffa0daf4d8>] ? mdt_obd_disconnect+0x48/0x4d0 [mdt]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.674236]  [<ffffffffa03ee2d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.694730]  [<ffffffffa05462d8>] ? class_fail_export+0x248/0x580 [obdclass]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.723473]  [<ffffffffa06e9473>] ? expired_lock_main+0x483/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.743981]  [<ffffffff81063b80>] ? default_wake_function+0x0/0x20
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.764186]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.784345]  [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.804021]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.824375]  [<ffffffffa06e8ff0>] ? expired_lock_main+0x0/0x830 [ptlrpc]
Mar 31 23:59:11 atlas-tds-mds1 kernel: [35429.853086]  [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
