https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1938 Lustre <p>Newly deployed filesystem. Running initial test rsync from client to filesystem when rsync process hung.</p> <p>Also see other weirdness. Poor performance. Network errors (example below).</p> <hr /> <p>Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1347583367/real 1347583367&#93;</span> req@ffff880664530c00 x1413043208599797/t0(0) o8-&gt;alicefs-OST0006-osc-ffff880832436000@172.20.2.125@o2ib:28/4 lens 368/512 e 0 to 1 dl 1347583422 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 5 previous similar messages<br/> Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1347583517/real 1347583517&#93;</span> req@ffff8808329b1800 x1413043208681668/t0(0) o8-&gt;alicefs-OST0006-osc-ffff880832436000@172.20.2.125@o2ib:28/4 lens 368/512 e 0 to 1 dl 1347583572 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: 16691:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 11 previous similar messages</p> <hr /> <p>INFO: task rsync:16826 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> rsync D 0000000000000002 0 16826 1 0x00000080<br/> ffff8803df615978 0000000000000046 0000000000000000 ffff88043343bca0<br/> ffff8808340ecb58 ffff880833270b40 ffff88043343bd38 ffff8803df615938<br/> ffff88043405b058 ffff8803df615fd8 000000000000fb88 ffff88043405b058<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa04c067e&gt;&#93;</span> cfs_waitq_wait+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a6c314&gt;&#93;</span> lov_delete_raid0+0x2e4/0x400 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a6b816&gt;&#93;</span> lov_object_delete+0x66/0x190 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e2d29&gt;&#93;</span> lu_object_free+0x89/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ce894&gt;&#93;</span> ? cfs_hash_dual_bd_unlock+0x34/0x60 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04cf052&gt;&#93;</span> ? cfs_hash_bd_from_key+0x42/0xe0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e300f&gt;&#93;</span> lu_object_put+0x1bf/0x290 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b4c312&gt;&#93;</span> ? cl_inode_fini+0x52/0x270 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e9a1e&gt;&#93;</span> cl_object_put+0xe/0x10 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b4c34a&gt;&#93;</span> cl_inode_fini+0x8a/0x270 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c1cbec&gt;&#93;</span> ? lmv_change_cbdata+0x2cc/0x580 <span class="error">&#91;lmv&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b1c42f&gt;&#93;</span> ll_clear_inode+0x2ff/0xdb0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81195c0c&gt;&#93;</span> clear_inode+0xac/0x140<br/> <span class="error">&#91;&lt;ffffffffa0b1747b&gt;&#93;</span> ll_delete_inode+0x18b/0x650 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b172f0&gt;&#93;</span> ? ll_delete_inode+0x0/0x650 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8119630e&gt;&#93;</span> generic_delete_inode+0xde/0x1d0<br/> <span class="error">&#91;&lt;ffffffff81196465&gt;&#93;</span> generic_drop_inode+0x65/0x80<br/> <span class="error">&#91;&lt;ffffffff811952b2&gt;&#93;</span> iput+0x62/0x70<br/> <span class="error">&#91;&lt;ffffffffa0af38d1&gt;&#93;</span> ll_d_iput+0x31/0x80 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81191dfc&gt;&#93;</span> dentry_iput+0x7c/0x100<br/> <span class="error">&#91;&lt;ffffffff81191f71&gt;&#93;</span> d_kill+0x31/0x60<br/> <span class="error">&#91;&lt;ffffffff8119399c&gt;&#93;</span> dput+0x7c/0x150<br/> <span class="error">&#91;&lt;ffffffff8117caf9&gt;&#93;</span> __fput+0x189/0x210<br/> <span class="error">&#91;&lt;ffffffff8117cba5&gt;&#93;</span> fput+0x25/0x30<br/> <span class="error">&#91;&lt;ffffffff811785cd&gt;&#93;</span> filp_close+0x5d/0x90<br/> <span class="error">&#91;&lt;ffffffff8106e3bf&gt;&#93;</span> put_files_struct+0x7f/0xf0<br/> <span class="error">&#91;&lt;ffffffff8106e483&gt;&#93;</span> exit_files+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff810704fd&gt;&#93;</span> do_exit+0x18d/0x870<br/> <span class="error">&#91;&lt;ffffffff810d6b12&gt;&#93;</span> ? audit_syscall_entry+0x272/0x2a0<br/> <span class="error">&#91;&lt;ffffffff81070c38&gt;&#93;</span> do_group_exit+0x58/0xd0<br/> <span class="error">&#91;&lt;ffffffff81070cc7&gt;&#93;</span> sys_exit_group+0x17/0x20<br/> <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> INFO: task rsync:16826 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> rsync D 0000000000000002 0 16826 1 0x00000080<br/> ffff8803df615978 0000000000000046 0000000000000000 ffff88043343bca0<br/> ffff8808340ecb58 ffff880833270b40 ffff88043343bd38 ffff8803df615938<br/> ffff88043405b058 ffff8803df615fd8 000000000000fb88 ffff88043405b058<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa04c067e&gt;&#93;</span> cfs_waitq_wait+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a6c314&gt;&#93;</span> lov_delete_raid0+0x2e4/0x400 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a6b816&gt;&#93;</span> lov_object_delete+0x66/0x190 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e2d29&gt;&#93;</span> lu_object_free+0x89/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ce894&gt;&#93;</span> ? cfs_hash_dual_bd_unlock+0x34/0x60 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04cf052&gt;&#93;</span> ? cfs_hash_bd_from_key+0x42/0xe0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e300f&gt;&#93;</span> lu_object_put+0x1bf/0x290 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b4c312&gt;&#93;</span> ? cl_inode_fini+0x52/0x270 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e9a1e&gt;&#93;</span> cl_object_put+0xe/0x10 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b4c34a&gt;&#93;</span> cl_inode_fini+0x8a/0x270 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c1cbec&gt;&#93;</span> ? lmv_change_cbdata+0x2cc/0x580 <span class="error">&#91;lmv&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b1c42f&gt;&#93;</span> ll_clear_inode+0x2ff/0xdb0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81195c0c&gt;&#93;</span> clear_inode+0xac/0x140<br/> <span class="error">&#91;&lt;ffffffffa0b1747b&gt;&#93;</span> ll_delete_inode+0x18b/0x650 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b172f0&gt;&#93;</span> ? ll_delete_inode+0x0/0x650 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8119630e&gt;&#93;</span> generic_delete_inode+0xde/0x1d0<br/> <span class="error">&#91;&lt;ffffffff81196465&gt;&#93;</span> generic_drop_inode+0x65/0x80<br/> <span class="error">&#91;&lt;ffffffff811952b2&gt;&#93;</span> iput+0x62/0x70<br/> <span class="error">&#91;&lt;ffffffffa0af38d1&gt;&#93;</span> ll_d_iput+0x31/0x80 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81191dfc&gt;&#93;</span> dentry_iput+0x7c/0x100<br/> <span class="error">&#91;&lt;ffffffff81191f71&gt;&#93;</span> d_kill+0x31/0x60<br/> <span class="error">&#91;&lt;ffffffff8119399c&gt;&#93;</span> dput+0x7c/0x150<br/> <span class="error">&#91;&lt;ffffffff8117caf9&gt;&#93;</span> __fput+0x189/0x210<br/> <span class="error">&#91;&lt;ffffffff8117cba5&gt;&#93;</span> fput+0x25/0x30<br/> <span class="error">&#91;&lt;ffffffff811785cd&gt;&#93;</span> filp_close+0x5d/0x90<br/> <span class="error">&#91;&lt;ffffffff8106e3bf&gt;&#93;</span> put_files_struct+0x7f/0xf0<br/> <span class="error">&#91;&lt;ffffffff8106e483&gt;&#93;</span> exit_files+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff810704fd&gt;&#93;</span> do_exit+0x18d/0x870<br/> <span class="error">&#91;&lt;ffffffff810d6b12&gt;&#93;</span> ? audit_syscall_entry+0x272/0x2a0<br/> <span class="error">&#91;&lt;ffffffff81070c38&gt;&#93;</span> do_group_exit+0x58/0xd0<br/> <span class="error">&#91;&lt;ffffffff81070cc7&gt;&#93;</span> sys_exit_group+0x17/0x20<br/> <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> RHEL 6.2, kernel-2.6.32-279.2.1.el6_x86_64, RHEL kernel_ib on MDS/OSS nodes, Mellanox OFED 1-5.3-3.1.0 on client.<br/> lustre-2.1.3 whamcloud rpms on MDS/OSS. lustre-client and lustre-client-modules v2.1.3 build from source against Mellanox OFED 1.5.3-3.1.0 LU-1938

Operational incompatibility - RHEL6.2, kernel-2.6.32-279.2.1.el6_x86_64, MLNX_OFED-1.5.3-3.1.0

Bug Major Resolved Not a Bug Doug Oucharek Jeff Johnson Fri, 14 Sep 2012 02:19:42 +0000 Sat, 8 Mar 2014 00:59:45 +0000 Sat, 8 Mar 2014 00:59:45 +0000 Lustre 2.1.3 0 5 <p>Is it necessary to run the same IB stack on client and server nodes? Server side is currently running RHEL 6.2 in kernel IB. Clients are running MLNX_OFED-1.5.3-3.1.0</p> <p>Doug will help with this one</p> <p>I have rebuild the server side rpms against the same Mellanox OFED release used on the client side. I will have the filesystem up in a few minutes and will report outcome. </p> <p>Kernel: 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64<br/> OFED: MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_6<br/> Lustre source: lustre-2.1.3-2.6.32_279.2.1.el6_lustre.gc46c389.x86_64.src.rpm</p> <p>It looked like the rebuild address the IB issues (lent_selftest works fine). Outstanding issues: rsync hangs after a few seconds and standard df hangs (but lfs df works).</p> <p>Jeff: did you get a chance to verify if rsync and dd are now working?</p> <p>Doug, I recompiled Lustre 2.1.3 source against Mellanox OFED 1.5.3-3.1.0. With the server and client side running the same IB stack it appears there are no problems.</p> <p>I had seen this once before in RHEL5.5, Lustre 1.8.4 where the client/cluster side ran OFED and the Lustre server side ran RHEL's in-kernel IB and the lnet communication was dysfunctional. As we know, standards aren't really that standard.</p> <p>I am going to test this on our in-house cluster when I get back to the US. I think there is a lingering incompatibility between RHEL's IB and clients running OFED. I will pass my findings along to WC.</p> <p>Since OFED is so popular on the cluster node side of the equation perhaps WC might consider doing an OFED variant of the precompiled Lustre packages. Just a thought. </p> <p>It could be hard to find out which official OFED version a RHEL kernel was based on. The kernel merge windows and OFED release cycles don't often match and very often Redhat would backport fixes from newer versions of OFED. I think it's a good idea for clients and servers to at least use IB stacks from a same source, e.g. Open Fabrics or Mellanox or Redhat - there's at least change logs to look for differences.</p> <p>Looks like this issue has been resolved and no further action is required.<br/> Thanks,<br/> ~ jfc.</p> Development Epic hang Rank 1|hzvzzj: Rank (Obsolete) 10076 Severity