Lustre: client enabled MDS capability!
Lustre: client enabled OSS capability!
Lustre: Mounted lustre-client
INFO: task ior:76875 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000000     0 76875  76869 0x00000000
 ffff88083f5ddd18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88083f5ddc88 ffffffff81227e9f ffff88083f5ddd68 ffffffff81199045
 ffff880871a9b058 ffff88083f5ddfd8 000000000000fbc8 ffff880871a9b058
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76877 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000004     0 76877  76869 0x00000000
 ffff88044926fd18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88044926fc88 ffffffff81227e9f ffffffff8100bb8e ffff88044926fd18
 ffff8808727be5f8 ffff88044926ffd8 000000000000fbc8 ffff8808727be5f8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff81055457>] ? mutex_spin_on_owner+0x87/0xc0
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76878 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000006     0 76878  76869 0x00000000
 ffff88046e7e3d18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88046e7e3c88 ffffffff81227e9f ffff88046e7e3d68 ffffffff81199045
 ffff880873679af8 ffff88046e7e3fd8 000000000000fbc8 ffff880873679af8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76881 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 000000000000000d     0 76881  76869 0x00000000
 ffff88082fe63d18 0000000000000086 0000000000000000 ffff8808341ab1d8
 ffff88082fe63c88 ffffffff81227e9f ffff88082fe63d68 ffffffff81199045
 ffff8808715ed058 ffff88082fe63fd8 000000000000fbc8 ffff8808715ed058
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76875 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000000     0 76875  76869 0x00000000
 ffff88083f5ddd18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88083f5ddc88 ffffffff81227e9f ffff88083f5ddd68 ffffffff81199045
 ffff880871a9b058 ffff88083f5ddfd8 000000000000fbc8 ffff880871a9b058
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76877 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000004     0 76877  76869 0x00000000
 ffff88044926fd18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88044926fc88 ffffffff81227e9f ffff88044926fd68 ffffffff81199045
 ffff8808727be5f8 ffff88044926ffd8 000000000000fbc8 ffff8808727be5f8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76878 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000007     0 76878  76869 0x00000000
 ffff88046e7e3d18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff88046e7e3c88 ffffffff81227e9f ffff88046e7e3d68 ffffffff81199045
 ffff880873679af8 ffff88046e7e3fd8 000000000000fbc8 ffff880873679af8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76879 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 0000000000000008     0 76879  76869 0x00000000
 ffff880824d13d18 0000000000000082 0000000000000000 ffff8808341ab1d8
 ffff880824d13c88 ffffffff81227e9f ffff880824d13d68 ffffffff81199045
 ffff880873679098 ffff880824d13fd8 000000000000fbc8 ffff880873679098
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76880 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 000000000000000b     0 76880  76869 0x00000000
 ffff880828827d18 0000000000000086 0000000000000000 ffff8808341ab1d8
 ffff880828827c88 ffffffff81227e9f ffff880828827d68 ffffffff81199045
 ffff8808715ec5f8 ffff880828827fd8 000000000000fbc8 ffff8808715ec5f8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ior:76882 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ior           D 000000000000000f     0 76882  76869 0x00000000
 ffff880826713d18 0000000000000086 0000000000000000 ffff8808341ab1d8
 ffff880826713c88 ffffffff81227e9f ffff880826713d68 ffffffff81199045
 ffff880871a845f8 ffff880826713fd8 000000000000fbc8 ffff880871a845f8
Call Trace:
 [<ffffffff81227e9f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff81199045>] ? __link_path_walk+0x145/0x1000
 [<ffffffff8152a5be>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff8152a45b>] mutex_lock+0x2b/0x50
 [<ffffffff8119ba76>] do_filp_open+0x2d6/0xd20
 [<ffffffff811bd6b8>] ? do_statfs_native+0x98/0xb0
 [<ffffffff8128f83a>] ? strncpy_from_user+0x4a/0x90
 [<ffffffff811a8b82>] ? alloc_fd+0x92/0x160
 [<ffffffff81185be9>] do_sys_open+0x69/0x140
 [<ffffffff81185d00>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
LustreError: 11-0: lustre-OST0024-osc-ffff880433184800: operation obd_ping to node 192.168.120.36@o2ib failed: rc = -107
Lustre: lustre-OST0024-osc-ffff880433184800: Connection to lustre-OST0024 (at 192.168.120.36@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0024-osc-ffff880433184800: This client was evicted by lustre-OST0024; in progress operations using this service will fail.
LustreError: 99105:0:(ldlm_resource.c:777:ldlm_resource_complain()) lustre-OST0024-osc-ffff880433184800: namespace resource [0x1d8c6:0x0:0x0].0 (ffff880872a6cd00) refcount nonzero (2) after lock cleanup; forcing cleanup.
LustreError: 99105:0:(ldlm_resource.c:1370:ldlm_resource_dump()) --- Resource: [0x1d8c6:0x0:0x0].0 (ffff880872a6cd00) refcount = 5
LustreError: 99105:0:(ldlm_resource.c:1373:ldlm_resource_dump()) Granted locks (in reverse order):
LustreError: 99105:0:(ldlm_resource.c:1376:ldlm_resource_dump()) ### ### ns: lustre-OST0024-osc-ffff880433184800 lock: ffff880f2871cb00/0x42e28ecfaef2b33a lrc: 3/0,0 mode: PR/PR res: [0x1d8c6:0x0:0x0].0 rrc: 5 type: EXT [0->18446744073709551615] (req 29796335616->29930553343) flags: 0x52f400000000 nid: local remote: 0x9c91b8d7046afd8 expref: -99 pid: 99045 timeout: 0 lvb_type: 1
Lustre: lustre-OST0024-osc-ffff880433184800: Connection restored to lustre-OST0024 (at 192.168.120.36@o2ib)
LustreError: 11-0: lustre-OST002c-osc-ffff880433184800: operation obd_ping to node 192.168.120.39@o2ib failed: rc = -107
Lustre: lustre-OST002c-osc-ffff880433184800: Connection to lustre-OST002c (at 192.168.120.39@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST002c-osc-ffff880433184800: This client was evicted by lustre-OST002c; in progress operations using this service will fail.
LustreError: 99113:0:(ldlm_resource.c:777:ldlm_resource_complain()) lustre-OST002c-osc-ffff880433184800: namespace resource [0x2163f:0x0:0x0].0 (ffff880859c76840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 99113:0:(ldlm_resource.c:1370:ldlm_resource_dump()) --- Resource: [0x2163f:0x0:0x0].0 (ffff880859c76840) refcount = 3
LustreError: 99113:0:(ldlm_resource.c:1373:ldlm_resource_dump()) Granted locks (in reverse order):
LustreError: 99113:0:(ldlm_resource.c:1376:ldlm_resource_dump()) ### ### ns: lustre-OST002c-osc-ffff880433184800 lock: ffff88106d6aba80/0x42e28ecfaef2b6ba lrc: 3/0,0 mode: PR/PR res: [0x2163f:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 29527900160->29662117887) flags: 0x52f400000000 nid: local remote: 0xcf3a80ba42247617 expref: -99 pid: 99044 timeout: 0 lvb_type: 1
LustreError: 99113:0:(ldlm_resource.c:1376:ldlm_resource_dump()) Skipped 1 previous similar message
Lustre: lustre-OST002c-osc-ffff880433184800: Connection restored to lustre-OST002c (at 192.168.120.39@o2ib)
LustreError: 11-0: lustre-OST0010-osc-ffff880433184800: operation obd_ping to node 192.168.120.37@o2ib failed: rc = -107
Lustre: lustre-OST0010-osc-ffff880433184800: Connection to lustre-OST0010 (at 192.168.120.37@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0010-osc-ffff880433184800: This client was evicted by lustre-OST0010; in progress operations using this service will fail.
LustreError: 99191:0:(ldlm_resource.c:777:ldlm_resource_complain()) lustre-OST0010-osc-ffff880433184800: namespace resource [0x25c37:0x0:0x0].0 (ffff880cef042d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 99191:0:(ldlm_resource.c:1370:ldlm_resource_dump()) --- Resource: [0x25c37:0x0:0x0].0 (ffff880cef042d00) refcount = 3
LustreError: 99191:0:(ldlm_resource.c:1373:ldlm_resource_dump()) Granted locks (in reverse order):
LustreError: 99191:0:(ldlm_resource.c:1376:ldlm_resource_dump()) ### ### ns: lustre-OST0010-osc-ffff880433184800 lock: ffff880f1a1afdc0/0x42e28ecfaef2bd74 lrc: 3/0,0 mode: PR/PR res: [0x25c37:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 29796335616->29930553343) flags: 0x52f400000000 nid: local remote: 0x7b6c600cbc16c38 expref: -99 pid: 99045 timeout: 0 lvb_type: 1
Lustre: lustre-OST0010-osc-ffff880433184800: Connection restored to lustre-OST0010 (at 192.168.120.37@o2ib)
LustreError: 11-0: lustre-OST002d-osc-ffff880433184800: operation obd_ping to node 192.168.120.34@o2ib failed: rc = -107
Lustre: lustre-OST002d-osc-ffff880433184800: Connection to lustre-OST002d (at 192.168.120.34@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST002d-osc-ffff880433184800: This client was evicted by lustre-OST002d; in progress operations using this service will fail.
LustreError: 99271:0:(ldlm_resource.c:777:ldlm_resource_complain()) lustre-OST002d-osc-ffff880433184800: namespace resource [0x28183:0x0:0x0].0 (ffff880cef0421c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 99271:0:(ldlm_resource.c:1370:ldlm_resource_dump()) --- Resource: [0x28183:0x0:0x0].0 (ffff880cef0421c0) refcount = 3
LustreError: 99271:0:(ldlm_resource.c:1373:ldlm_resource_dump()) Granted locks (in reverse order):
LustreError: 99271:0:(ldlm_resource.c:1376:ldlm_resource_dump()) ### ### ns: lustre-OST002d-osc-ffff880433184800 lock: ffff881016f56a80/0x42e28ecfaef2c102 lrc: 3/0,0 mode: PR/PR res: [0x28183:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 29527900160->29662117887) flags: 0x52f400000000 nid: local remote: 0xc2e6f2e60c6a3a1f expref: -99 pid: 99044 timeout: 0 lvb_type: 1
Lustre: lustre-OST002d-osc-ffff880433184800: Connection restored to lustre-OST002d (at 192.168.120.34@o2ib)