[LU-1938] Operational incompatibility - RHEL6.2, kernel-2.6.32-279.2.1.el6_x86_64, MLNX_OFED-1.5.3-3.1.0 - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Not a Bug
Priority: Major
Fix Version/s: None
Affects Version/s: Lustre 2.1.3
Labels:
None
Environment:

Hide
RHEL 6.2, kernel-2.6.32-279.2.1.el6_x86_64, RHEL kernel_ib on MDS/OSS nodes, Mellanox OFED 1-5.3-3.1.0 on client.
lustre-2.1.3 whamcloud rpms on MDS/OSS. lustre-client and lustre-client-modules v2.1.3 build from source against Mellanox OFED 1.5.3-3.1.0

Show
RHEL 6.2, kernel-2.6.32-279.2.1.el6_x86_64, RHEL kernel_ib on MDS/OSS nodes, Mellanox OFED 1-5.3-3.1.0 on client. lustre-2.1.3 whamcloud rpms on MDS/OSS. lustre-client and lustre-client-modules v2.1.3 build from source against Mellanox OFED 1.5.3-3.1.0

Severity:
3
Epic:
- hang
Rank (Obsolete):
10076

Description

Newly deployed filesystem. Running initial test rsync from client to filesystem when rsync process hung.

Also see other weirdness. Poor performance. Network errors (example below).

Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1347583367/real 1347583367] req@ffff880664530c00 x1413043208599797/t0(0) o8->alicefs-OST0006-osc-ffff880832436000@172.20.2.125@o2ib:28/4 lens 368/512 e 0 to 1 dl 1347583422 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1347583517/real 1347583517] req@ffff8808329b1800 x1413043208681668/t0(0) o8->alicefs-OST0006-osc-ffff880832436000@172.20.2.125@o2ib:28/4 lens 368/512 e 0 to 1 dl 1347583572 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 11 previous similar messages

INFO: task rsync:16826 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsync D 0000000000000002 0 16826 1 0x00000080
ffff8803df615978 0000000000000046 0000000000000000 ffff88043343bca0
ffff8808340ecb58 ffff880833270b40 ffff88043343bd38 ffff8803df615938
ffff88043405b058 ffff8803df615fd8 000000000000fb88 ffff88043405b058
Call Trace:
[<ffffffffa04c067e>] cfs_waitq_wait+0xe/0x10 [libcfs]
[<ffffffffa0a6c314>] lov_delete_raid0+0x2e4/0x400 [lov]
[<ffffffffa0a6b816>] lov_object_delete+0x66/0x190 [lov]
[<ffffffffa05e2d29>] lu_object_free+0x89/0x1b0 [obdclass]
[<ffffffffa04ce894>] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs]
[<ffffffffa04cf052>] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs]
[<ffffffffa05e300f>] lu_object_put+0x1bf/0x290 [obdclass]
[<ffffffffa0b4c312>] ? cl_inode_fini+0x52/0x270 [lustre]
[<ffffffffa05e9a1e>] cl_object_put+0xe/0x10 [obdclass]
[<ffffffffa0b4c34a>] cl_inode_fini+0x8a/0x270 [lustre]
[<ffffffffa0c1cbec>] ? lmv_change_cbdata+0x2cc/0x580 [lmv]
[<ffffffffa0b1c42f>] ll_clear_inode+0x2ff/0xdb0 [lustre]
[<ffffffff81195c0c>] clear_inode+0xac/0x140
[<ffffffffa0b1747b>] ll_delete_inode+0x18b/0x650 [lustre]
[<ffffffffa0b172f0>] ? ll_delete_inode+0x0/0x650 [lustre]
[<ffffffff8119630e>] generic_delete_inode+0xde/0x1d0
[<ffffffff81196465>] generic_drop_inode+0x65/0x80
[<ffffffff811952b2>] iput+0x62/0x70
[<ffffffffa0af38d1>] ll_d_iput+0x31/0x80 [lustre]
[<ffffffff81191dfc>] dentry_iput+0x7c/0x100
[<ffffffff81191f71>] d_kill+0x31/0x60
[<ffffffff8119399c>] dput+0x7c/0x150
[<ffffffff8117caf9>] __fput+0x189/0x210
[<ffffffff8117cba5>] fput+0x25/0x30
[<ffffffff811785cd>] filp_close+0x5d/0x90
[<ffffffff8106e3bf>] put_files_struct+0x7f/0xf0
[<ffffffff8106e483>] exit_files+0x53/0x70
[<ffffffff810704fd>] do_exit+0x18d/0x870
[<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff81070c38>] do_group_exit+0x58/0xd0
[<ffffffff81070cc7>] sys_exit_group+0x17/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task rsync:16826 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsync D 0000000000000002 0 16826 1 0x00000080
ffff8803df615978 0000000000000046 0000000000000000 ffff88043343bca0
ffff8808340ecb58 ffff880833270b40 ffff88043343bd38 ffff8803df615938
ffff88043405b058 ffff8803df615fd8 000000000000fb88 ffff88043405b058
Call Trace:
[<ffffffffa04c067e>] cfs_waitq_wait+0xe/0x10 [libcfs]
[<ffffffffa0a6c314>] lov_delete_raid0+0x2e4/0x400 [lov]
[<ffffffffa0a6b816>] lov_object_delete+0x66/0x190 [lov]
[<ffffffffa05e2d29>] lu_object_free+0x89/0x1b0 [obdclass]
[<ffffffffa04ce894>] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs]
[<ffffffffa04cf052>] ? cfs_hash_bd_from_key+0x42/0xe0 [libcfs]
[<ffffffffa05e300f>] lu_object_put+0x1bf/0x290 [obdclass]
[<ffffffffa0b4c312>] ? cl_inode_fini+0x52/0x270 [lustre]
[<ffffffffa05e9a1e>] cl_object_put+0xe/0x10 [obdclass]
[<ffffffffa0b4c34a>] cl_inode_fini+0x8a/0x270 [lustre]
[<ffffffffa0c1cbec>] ? lmv_change_cbdata+0x2cc/0x580 [lmv]
[<ffffffffa0b1c42f>] ll_clear_inode+0x2ff/0xdb0 [lustre]
[<ffffffff81195c0c>] clear_inode+0xac/0x140
[<ffffffffa0b1747b>] ll_delete_inode+0x18b/0x650 [lustre]
[<ffffffffa0b172f0>] ? ll_delete_inode+0x0/0x650 [lustre]
[<ffffffff8119630e>] generic_delete_inode+0xde/0x1d0
[<ffffffff81196465>] generic_drop_inode+0x65/0x80
[<ffffffff811952b2>] iput+0x62/0x70
[<ffffffffa0af38d1>] ll_d_iput+0x31/0x80 [lustre]
[<ffffffff81191dfc>] dentry_iput+0x7c/0x100
[<ffffffff81191f71>] d_kill+0x31/0x60
[<ffffffff8119399c>] dput+0x7c/0x150
[<ffffffff8117caf9>] __fput+0x189/0x210
[<ffffffff8117cba5>] fput+0x25/0x30
[<ffffffff811785cd>] filp_close+0x5d/0x90
[<ffffffff8106e3bf>] put_files_struct+0x7f/0xf0
[<ffffffff8106e483>] exit_files+0x53/0x70
[<ffffffff810704fd>] do_exit+0x18d/0x870
[<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff81070c38>] do_group_exit+0x58/0xd0
[<ffffffff81070cc7>] sys_exit_group+0x17/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b

Attachments

Activity

[LU-1938] Operational incompatibility - RHEL6.2, kernel-2.6.32-279.2.1.el6_x86_64, MLNX_OFED-1.5.3-3.1.0

Jeff Johnson (Inactive) added a comment - 18/Sep/12 10:55 AM

Doug, I recompiled Lustre 2.1.3 source against Mellanox OFED 1.5.3-3.1.0. With the server and client side running the same IB stack it appears there are no problems.

I had seen this once before in RHEL5.5, Lustre 1.8.4 where the client/cluster side ran OFED and the Lustre server side ran RHEL's in-kernel IB and the lnet communication was dysfunctional. As we know, standards aren't really that standard.

I am going to test this on our in-house cluster when I get back to the US. I think there is a lingering incompatibility between RHEL's IB and clients running OFED. I will pass my findings along to WC.

Since OFED is so popular on the cluster node side of the equation perhaps WC might consider doing an OFED variant of the precompiled Lustre packages. Just a thought.

Jeff Johnson (Inactive) added a comment - 18/Sep/12 10:55 AM Doug, I recompiled Lustre 2.1.3 source against Mellanox OFED 1.5.3-3.1.0. With the server and client side running the same IB stack it appears there are no problems. I had seen this once before in RHEL5.5, Lustre 1.8.4 where the client/cluster side ran OFED and the Lustre server side ran RHEL's in-kernel IB and the lnet communication was dysfunctional. As we know, standards aren't really that standard. I am going to test this on our in-house cluster when I get back to the US. I think there is a lingering incompatibility between RHEL's IB and clients running OFED. I will pass my findings along to WC. Since OFED is so popular on the cluster node side of the equation perhaps WC might consider doing an OFED variant of the precompiled Lustre packages. Just a thought.

Doug Oucharek (Inactive) added a comment - 17/Sep/12 7:43 PM

Jeff: did you get a chance to verify if rsync and dd are now working?

Doug Oucharek (Inactive) added a comment - 17/Sep/12 7:43 PM Jeff: did you get a chance to verify if rsync and dd are now working?

Doug Oucharek (Inactive) added a comment - 14/Sep/12 4:44 PM

It looked like the rebuild address the IB issues (lent_selftest works fine). Outstanding issues: rsync hangs after a few seconds and standard df hangs (but lfs df works).

Doug Oucharek (Inactive) added a comment - 14/Sep/12 4:44 PM It looked like the rebuild address the IB issues (lent_selftest works fine). Outstanding issues: rsync hangs after a few seconds and standard df hangs (but lfs df works).

Jeff Johnson (Inactive) added a comment - 14/Sep/12 2:56 PM

I have rebuild the server side rpms against the same Mellanox OFED release used on the client side. I will have the filesystem up in a few minutes and will report outcome.

Kernel: 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64
OFED: MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_6
Lustre source: lustre-2.1.3-2.6.32_279.2.1.el6_lustre.gc46c389.x86_64.src.rpm

Jeff Johnson (Inactive) added a comment - 14/Sep/12 2:56 PM I have rebuild the server side rpms against the same Mellanox OFED release used on the client side. I will have the filesystem up in a few minutes and will report outcome. Kernel: 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 OFED: MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_6 Lustre source: lustre-2.1.3-2.6.32_279.2.1.el6_lustre.gc46c389.x86_64.src.rpm

Peter Jones added a comment - 14/Sep/12 2:48 PM

Doug will help with this one

Peter Jones added a comment - 14/Sep/12 2:48 PM Doug will help with this one

Jeff Johnson (Inactive) added a comment - 14/Sep/12 2:21 AM

Is it necessary to run the same IB stack on client and server nodes? Server side is currently running RHEL 6.2 in kernel IB. Clients are running MLNX_OFED-1.5.3-3.1.0

Jeff Johnson (Inactive) added a comment - 14/Sep/12 2:21 AM Is it necessary to run the same IB stack on client and server nodes? Server side is currently running RHEL 6.2 in kernel IB. Clients are running MLNX_OFED-1.5.3-3.1.0

People

Assignee:: Doug Oucharek (Inactive)

Reporter:: Jeff Johnson (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 14/Sep/12 2:19 AM

Updated:: 08/Mar/14 12:59 AM

Resolved:: 08/Mar/14 12:59 AM