https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3765 Lustre <p>sanity test 24u hit the following failure on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:06:31:Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 11:06:31 (1376417191) 11:06:31:LustreError: 13255:0:(mdt_handler.c:224:mdt_lock_pdo_init()) ASSERTION( namelen &gt; 0 ) failed: 11:06:31:LustreError: 13255:0:(mdt_handler.c:224:mdt_lock_pdo_init()) LBUG 11:06:31:Pid: 13255, comm: mdt_01 11:06:31: 11:06:31:Call Trace: 11:06:31: [&lt;ffffffffa04d0785&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 11:06:31: [&lt;ffffffffa04d0d97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 11:06:31: [&lt;ffffffffa0bdea65&gt;] mdt_lock_pdo_init+0xe5/0xf0 [mdt] 11:06:31: [&lt;ffffffffa0c127c6&gt;] mdt_reint_open+0x1f6/0x2940 [mdt] 11:06:31: [&lt;ffffffffa077b764&gt;] ? lustre_msg_add_version+0x74/0xd0 [ptlrpc] 11:06:32: [&lt;ffffffffa0ba256e&gt;] ? md_ucred+0x1e/0x60 [mdd] 11:06:32: [&lt;ffffffffa0be15d5&gt;] ? mdt_ucred+0x15/0x20 [mdt] 11:06:32: [&lt;ffffffffa0bf84ec&gt;] ? mdt_root_squash+0x2c/0x3e0 [mdt] 11:06:32: [&lt;ffffffffa0bfcc51&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 11:06:32: [&lt;ffffffffa0bf3ed4&gt;] mdt_reint_internal+0x544/0x8e0 [mdt] 11:06:32: [&lt;ffffffffa0bf453d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] 11:06:32: [&lt;ffffffffa0bf2c09&gt;] mdt_intent_policy+0x379/0x690 [mdt] 11:06:32: [&lt;ffffffffa0737391&gt;] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] 11:06:32: [&lt;ffffffffa075d1ed&gt;] ldlm_handle_enqueue0+0x48d/0xf50 [ptlrpc] 11:06:32: [&lt;ffffffffa0bf3586&gt;] mdt_enqueue+0x46/0x130 [mdt] 11:06:32: [&lt;ffffffffa0be8772&gt;] mdt_handle_common+0x932/0x1750 [mdt] 11:06:32: [&lt;ffffffffa0be9665&gt;] mdt_regular_handle+0x15/0x20 [mdt] 11:06:32: [&lt;ffffffffa078bbae&gt;] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] 11:06:32: [&lt;ffffffffa078af60&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 11:06:32: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 11:06:32: [&lt;ffffffffa078af60&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 11:06:32: [&lt;ffffffffa078af60&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 11:06:32: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 11:06:32: 11:06:32:Kernel panic - not syncing: LBUG </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/e3f3b3d8-0525-11e3-8d88-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e3f3b3d8-0525-11e3-8d88-52540035b04c</a></p> <p>More instances:<br/> <a href="https://maloo.whamcloud.com/test_sets/369e054c-0059-11e3-bb00-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/369e054c-0059-11e3-bb00-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/0bf3fdbc-f8f5-11e2-8917-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0bf3fdbc-f8f5-11e2-8917-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/59b2a818-f504-11e2-a8f6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/59b2a818-f504-11e2-a8f6-52540035b04c</a></p> <br/> Lustre client build: <a href="http://build.whamcloud.com/job/lustre-master/1613/">http://build.whamcloud.com/job/lustre-master/1613/</a><br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_1/191/">http://build.whamcloud.com/job/lustre-b2_1/191/</a> (2.1.5)<br/> Distro/Arch: RHEL6.4/x86_64<br/> LU-3765

2.5.0<->2.1.5 interop: sanity test 24u: (mdt_handler.c:224:mdt_lock_pdo_init()) ASSERTION( namelen > 0 ) failed

Bug Blocker Resolved Duplicate Lai Siyao Jian Yu HB mn1 Thu, 15 Aug 2013 15:05:49 +0000 Thu, 24 Oct 2013 19:03:30 +0000 Tue, 1 Oct 2013 16:37:37 +0000 Lustre 2.1.6 Lustre 2.5.0 0 13 <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b2_4/29/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/29/</a><br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_1/215/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/215/</a> (2.1.6)<br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The sanity test 24u also hung and hit the same LBUG on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:33:17:Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 08:33:10 (1376407990) 08:33:17:LustreError: 13776:0:(mdt_handler.c:224:mdt_lock_pdo_init()) ASSERTION( namelen &gt; 0 ) failed: 08:33:17:LustreError: 13776:0:(mdt_handler.c:224:mdt_lock_pdo_init()) LBUG 08:33:17:Pid: 13776, comm: mdt_00 08:33:18: 08:33:18:Call Trace: 08:33:18: [&lt;ffffffffa0472785&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 08:33:18: [&lt;ffffffffa0472d97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 08:33:18: [&lt;ffffffffa0b6ea65&gt;] mdt_lock_pdo_init+0xe5/0xf0 [mdt] 08:33:18: [&lt;ffffffffa0ba28a6&gt;] mdt_reint_open+0x1f6/0x2940 [mdt] 08:33:18: [&lt;ffffffffa0715754&gt;] ? lustre_msg_add_version+0x74/0xd0 [ptlrpc] 08:33:18: [&lt;ffffffffa0b3356e&gt;] ? md_ucred+0x1e/0x60 [mdd] 08:33:18: [&lt;ffffffffa0b715d5&gt;] ? mdt_ucred+0x15/0x20 [mdt] 08:33:18: [&lt;ffffffffa0b884ec&gt;] ? mdt_root_squash+0x2c/0x3e0 [mdt] 08:33:18: [&lt;ffffffffa0b8cc51&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 08:33:18: [&lt;ffffffffa0b83ed4&gt;] mdt_reint_internal+0x544/0x8e0 [mdt] 08:33:18: [&lt;ffffffffa0b8453d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] 08:33:18: [&lt;ffffffffa0b82c09&gt;] mdt_intent_policy+0x379/0x690 [mdt] 08:33:18: [&lt;ffffffffa06d1391&gt;] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] 08:33:18: [&lt;ffffffffa06f71dd&gt;] ldlm_handle_enqueue0+0x48d/0xf50 [ptlrpc] 08:33:18: [&lt;ffffffffa0b83586&gt;] mdt_enqueue+0x46/0x130 [mdt] 08:33:18: [&lt;ffffffffa0b78772&gt;] mdt_handle_common+0x932/0x1750 [mdt] 08:33:18: [&lt;ffffffffa0b79665&gt;] mdt_regular_handle+0x15/0x20 [mdt] 08:33:18: [&lt;ffffffffa0725b9e&gt;] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] 08:33:18: [&lt;ffffffffa0724f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 08:33:18: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 08:33:18: [&lt;ffffffffa0724f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 08:33:18: [&lt;ffffffffa0724f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] 08:33:18: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 08:33:18: 08:33:19:Kernel panic - not syncing: LBUG </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/ae6aa048-0569-11e3-b127-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ae6aa048-0569-11e3-b127-52540035b04c</a></p> <p>The same test passed on Lustre 2.4.0 client with 2.1.5 server:<br/> <a href="https://maloo.whamcloud.com/sub_tests/397da174-c63d-11e2-ad5d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/397da174-c63d-11e2-ad5d-52540035b04c</a></p> <p>This is a regression issue on Lustre b2_4 branch.</p> <p>This is clearly caused by </p> <p> <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a> nfs: writing to new files will return ENOENT</p> <p> This happend with SLES11SP2 Lustre client, which in turn acts as an<br/> NFS server, exporting a subtree of an Lustre fs through NFS.</p> <p> We detected that whenever we are writing to a new file using, fx,<br/> 'echo blah &gt; newfile', it will return ENOENT error. We found<br/> out that this was caused by the anonymous dentry. In SLESS11SP2,<br/> anonymous dentries are assigned '/' as the name, instead of an<br/> empty string. When MDT handles the intent_open call, it will look<br/> up the obj by the name if it is not an empty string, and thus<br/> couldn't find it.</p> <p> As MDS_OPEN_BY_FID is always set on this request, we never need<br/> to send the name in this request. The fid is already available<br/> and should be used in case the file has been renamed.</p> <p> Signed-off-by: Cheng Shao &lt;cheng_shao@xyratex.com&gt;<br/> Signed-off-by: Patrick Farrell &lt;paf@cray.com&gt;<br/> Change-Id: Ia8bd6f2814d05350d0a197df8a3ffd9729e2081b<br/> Reviewed-on: <a href="http://review.whamcloud.com/6920" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6920</a><br/> Reviewed-by: Bob Glossman &lt;bob.glossman@intel.com&gt;<br/> Tested-by: Hudson<br/> Reviewed-by: Alexey Shvetsov &lt;alexxy@gentoo.org&gt;<br/> Reviewed-by: Lai Siyao &lt;lai.siyao@intel.com&gt;<br/> Tested-by: Maloo &lt;whamcloud.maloo@gmail.com&gt;<br/> Reviewed-by: James Simmons &lt;uja.ornl@gmail.com&gt;<br/> Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt;</p> <p>In this patch, it stops sending name for open by FID and set lovea(test 24u) request. But 2.1.5 server can not handle<br/> this correctly. So we either</p> <p>1. fix 2.1.5 server to handle this zero name length issue. please check open_by_fid part in mdt_reint_open.<br/> 2. or fix b2_4 client to add open lock flag for lovea setting req, which can avoid the problem as well, IMHO. </p> <p>Lai</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>peter</p> <p>This is blocking the whole test session on Lustre b2_4 client with 2.1.6 server:<br/> <a href="https://maloo.whamcloud.com/test_sessions/ac905704-0569-11e3-b127-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/ac905704-0569-11e3-b127-52540035b04c</a></p> <p>IMO once client specified MDS_OPEN_BY_FID, MDS should never with open with name because name may be invalid, or it will cause inconsistency. If this is true, MDS open by fid code can be simplified a lot.</p> <p>Patch is on <a href="http://review.whamcloud.com/#/c/7358/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7358/</a></p> <p>Lai,</p> <p>Take a look at my latest in <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a>. I think it belongs there and not here, but it could go in either. It's about the problems with the proposed patch for <a href="https://jira.whamcloud.com/browse/LU-3765" title="2.5.0&lt;-&gt;2.1.5 interop: sanity test 24u: (mdt_handler.c:224:mdt_lock_pdo_init()) ASSERTION( namelen &gt; 0 ) failed" class="issue-link" data-issue-key="LU-3765"><del>LU-3765</del></a>.</p> <p>Patch for master is on:<br/> <a href="http://review.whamcloud.com/#/c/7475/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7475/</a><br/> <a href="http://review.whamcloud.com/#/c/7476/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7476/</a></p> <p>These patches enabled getattr/open-by-fid by default, thus either fid or name is packed in these requests, and server can handle op-by-fid correctly.</p> <p>Once these patches are accepted, they need to be backported to 2.4, and also fix 2.1 server code to maintain 2.5 &lt;-&gt; 2.1 interop. I'll continue working on this.</p> <p>We have reverted <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a> from b2_4 for now but are continuing to work on a more complete fix on master</p> <p>Patch for b2_1 is on: <a href="http://review.whamcloud.com/#/c/7627/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7627/</a></p> <p>Now it's ready to continue interop test between 2.5 and 2.1 with these three patches.</p> <p><a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a> was reverted from master as well</p> <p>This problem was introduced by the patches for <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a>, and is no longer an issue now that the patch has been reverted.</p> <p>The MDT is still generally at the mercy of the client to send valid names. Please see <a href="http://review.whamcloud.com/#/c/7961/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7961/</a> from <a href="https://jira.whamcloud.com/browse/LU-2875" title="Remove LASSERT()s on return values from req_capsule_client_get() and similar" class="issue-link" data-issue-key="LU-2875"><del>LU-2875</del></a>.</p> <p>Lai, for your comment below and the related patch here <a href="http://review.whamcloud.com/#/c/7627/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7627/</a>,</p> <blockquote> <p>Once these patches are accepted, they need to be backported to 2.4, and also fix 2.1 server code to maintain 2.5 &lt;-&gt; 2.1 interop. I'll continue working on this.</p></blockquote> <p>Vitaly has a different thought - we shouldn't make change in the old version server code to accommodate features introduced in the newer version. The necessary changes should be done solely in the newer version to fix any interop issues with old versions. I think that is a legitimate comment, although fixing the server side is much simpler in this case. Thoughts?</p> <p>While I agree with Vitaly in the abstract, I don't see how we can fix this issue on the 2.4 clients.</p> <p>The open_by_fid code on 2.1 is activated when the namelength is 0. SLES11SP2 changes anonymous dentries to be non-null/non-zero name length, so they no longer hit this check. We tried to fix this by having ll_intent_file_open not pass the name, but hit issues. The problem, as I see it, is that we need to only not send the name in the case of a root dentry.</p> <p>We tried that by using that code only when the DCACHE_DISCONNECTED flag was observed, but that still triggered the assertion on the MDS. It's not clear to me why - I thought DCACHE_DISCONNECTED was unique to the root dentry, but the observed behavior suggests not.</p> <p>The other possibility is that DCACHE_DISCONNECTED does uniquely identify the root dentry, but the zero-length root dentry from CentOS/SLES11SP1 comes across in ll_intent_file_open differently than just a 0 namelen and null pointer for the name, and that this difference is essential in avoiding the MDS crashes.</p> <p><span class="error">&#91;THIS INFORMATION IS INCORRECT. See my reply to Andreas below.&#93;</span><br/> So the problem remains: In order to achieve a 2.4/2.5 only fix, we must somehow force open_by_fid on 2.1 servers for root dentries, and only for root dentries. And we need to do it in a way that does not crash the 2.1 server, or cause other problems on 2.4 because of the nulled file name.<br/> <span class="error">&#91;^-- This is not only for root dentries, it is for all anonymous dentries.&#93;</span></p> <p>I'm not sure that's possible. That's why we chose to add MDS_OPEN_BY_FID to 2.1. This is a fairly minor patch to 2.1, as all it adds is this way to force 2.1 to do open_by_fid.</p> <p>FYI, DCACHE_DUSCONNECTED is used on any NFS inode that is not connected to the namespace (i.e. if NFS client does its own getattr-by-handle operation). To determine the root dentry you should check if dentry == sb-&gt;s_root. </p> <p>Andreas: Thanks for the reminder.</p> <p>Reading it again, I see my comment above is flawed. The change in newer kernels is all anonymous dentries, I wasn't thinking clearly when I wrote that. We don't need to identify merely the root dentries, this issue applies to all anonymous dentries.</p> <p>The problem is that when we change the names as we did, we fail that NAMELEN related assertion in 2.1. In retrospect, I think we have succeeded in our goal of isolating anonymous dentries, but perhaps there is a difference between passing NULL in to ll_prep_md_op_data for the name and the anonymous dentry names in 2.6.32 kernels. They appear (I haven't tested, but this is my reading of the code) to be pointers to a string containing nothing but the null terminator, but they aren't actually NULL.</p> <p>I wonder if this difference isn't significant.</p> Duplicate LU-3544 Related LU-2875 LU-3233 Development Rank 1|hzvy1j: Rank (Obsolete) 9699 Severity