https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11668 Lustre <p>I hit this assertion in current master-next testing but I don't see anything obvious included that would lead to it so perhaps it's some rare race that just happened to happen?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [ 6095.328424] Lustre: DEBUG MARKER: == racer test 1: racer on clients: centos-30.localnet DURATION=2700 ================================== 02:51:44 (1542181904) [ 6097.825252] Lustre: lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200000403:0x5:0x0], use llapi_layout_get_by_path() [ 6101.235171] Lustre: DEBUG MARKER: racer test_1: @@@@@@ FAIL: generate lss conf (mds1) [ 6106.472165] LustreError: 4856:0:(mdt_lvb.c:430:mdt_lvbo_fill()) lustre-MDT0000: small buffer size 448 for EA 496 (max_mdsize 496): rc = -34 [ 6108.575804] LustreError: 26511:0:(mdt_lvb.c:430:mdt_lvbo_fill()) lustre-MDT0001: small buffer size 448 for EA 472 (max_mdsize 472): rc = -34 [ 6361.959073] 9[28537]: segfault at 8 ip 00007f20a23dc958 sp 00007fffccffcf80 error 4 in ld-2.17.so[7f20a23d1000+22000] [ 6469.162820] LustreError: 26494:0:(mdd_dir.c:222:mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed: [ 6469.214647] LustreError: 26494:0:(mdd_dir.c:222:mdd_parent_fid()) LBUG [ 6469.215925] Pid: 26494, comm: mdt00_001 3.10.0-7.6-debug #1 SMP Wed Nov 7 21:55:08 EST 2018 [ 6469.219120] Call Trace: [ 6469.222463] [&lt;ffffffffa02637dc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 6469.250486] [&lt;ffffffffa026388c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 6469.251895] [&lt;ffffffffa100ef22&gt;] mdd_is_parent+0x4d2/0x510 [mdd] [ 6469.253469] [&lt;ffffffffa100f164&gt;] mdd_is_subdir+0x204/0x240 [mdd] [ 6469.315072] [&lt;ffffffffa108f8a0&gt;] mdt_reint_rename_internal.isra.47+0x810/0x2750 [mdt] [ 6469.318228] [&lt;ffffffffa109689b&gt;] mdt_reint_rename_or_migrate.isra.51+0x19b/0x860 [mdt] [ 6469.340401] [&lt;ffffffffa1096f93&gt;] mdt_reint_rename+0x13/0x20 [mdt] [ 6469.358495] [&lt;ffffffffa10984f0&gt;] mdt_reint_rec+0x80/0x210 [mdt] [ 6469.400446] [&lt;ffffffffa1075882&gt;] mdt_reint_internal+0x6b2/0xa50 [mdt] [ 6469.405016] [&lt;ffffffffa1080997&gt;] mdt_reint+0x67/0x140 [mdt] [ 6469.406310] [&lt;ffffffffa05c3365&gt;] tgt_request_handle+0xaf5/0x1590 [ptlrpc] [ 6469.412532] [&lt;ffffffffa0567436&gt;] ptlrpc_server_handle_request+0x256/0xad0 [ptlrpc] [ 6469.415111] [&lt;ffffffffa056b329&gt;] ptlrpc_main+0xa99/0x1f60 [ptlrpc] [ 6469.416569] [&lt;ffffffff810b4ed4&gt;] kthread+0xe4/0xf0 [ 6469.417870] [&lt;ffffffff817c4c77&gt;] ret_from_fork_nospec_end+0x0/0x39 [ 6469.419494] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [ 6469.420822] Kernel panic - not syncing: LBUG </pre> </div></div> <p>crashdump: 192.168.123.130-2018-11-14-02:58:09 git source: 46bcdb588e22abf162af9a486107c7b59b438dd2</p> LU-11668

mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) & 00170000) == 0040000) ) failed

Improvement Major Resolved Fixed Lai Siyao Oleg Drokin Wed, 14 Nov 2018 17:13:31 +0000 Sun, 1 Mar 2020 15:34:31 +0000 Thu, 29 Nov 2018 23:10:33 +0000 Lustre 2.12.0 Lustre 2.12.0 Lustre 2.14.0 0 6 <p>hit this twice more so far so it does appear to be a recent regression</p> <p>Lai</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>It may just be that this is a case of rename being called on a regular file and the MDS not verifying that the "parent" is a directory before diving into the code? This should probably be verified early on in RPC handing like <tt>mdt_reint_rename_internal()</tt> (or <tt>mdt_reint_rename_or_migrate()</tt> if we don't want to ever allow migrating regular files), but I couldn't find any checks like that.</p> <p>It would also be useful to improve the assertion here to tell us what the actual file type is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -219,7 +219,10 @@ <span class="code-keyword">static</span> inline <span class="code-object">int</span> mdd_parent_fid(<span class="code-keyword">const</span> struct lu_env *env, ENTRY; - LASSERT(S_ISDIR(mdd_object_type(obj))); + LASSERTF(S_ISDIR(mdd_object_type(obj)), + <span class="code-quote">"%s: FID "</span>DFID<span class="code-quote">" is not a directory type = %o\n"</span>, + mdd_obj_dev_name(obj), PFID(mdd_object_fid(obj)), + mdd_object_type(obj)); buf = lu_buf_check_and_alloc(buf, PATH_MAX); <span class="code-keyword">if</span> (buf-&gt;lb_buf == NULL) </pre> </div></div> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33700" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33700</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> debug: print object type in mdd_parent_fid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a4c91e284d998de40fd08bb18e046fb23bd0044d</p> <p>I don't find any clue from the code, let's see what type this object is. Since the parent FID is read from disk, and system may be inconsistent, in the future we may turn this assert into a check and return error if it's not directory.</p> <p>I think it makes sense to just check the parent type in the MDT code, since there could be all kinds of reasons that it is wrong. In this case, it is likely that racer moved or deleted a directory that the client was going to rename a file in, and another thread created a regular file in its place with the same name. The <tt>mdt_reint_rename_internal()</tt> code should just check the type after the parent is looked up, and return <tt>-ENOTDIR</tt> if it isn't a directory.</p> <p>It may be the best place for that is in <tt>mdt_object_find_check()</tt> since that is only called for parent directories, in which case it would be better to be renamed as <tt>mdt_parent_find_check()</tt> or similar.</p> <p>Could you please work on a patch today, as this is one of the last blockers for 2.12 that doesn't have a patch yet.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33709</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> mdt: check parent type in rename/migrate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3ede5bf5d6539ac2a8ca4831722bff367d1aed68</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33700/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33700/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> debug: print object type in mdd_parent_fid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9a4a99b81a267a098b92ec10991af27b7f3cae7e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33709/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33709/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> mdt: check parent type in rename/migrate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 214b12adc315c4adc3c56deb7e790fdc6f0095c8</p> <p>Landed for 2.12</p> <p>I hit this in current master-next with the new debug print. Running racer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5609.701558] LustreError: 29511:0:(mdd_dir.c:225:mdd_parent_fid()) ASSERTION( S_ISDIR(mdd_object_type(obj)) ) failed: lustre-MDD0000: FID [0x200000003:0xa:0x0] is not a directory type = 100000 [ 5609.713377] LustreError: 29511:0:(mdd_dir.c:225:mdd_parent_fid()) LBUG [ 5609.714440] Pid: 29511, comm: mdt07_012 3.10.0-7.6-debug #1 SMP Wed Nov 7 21:55:08 EST 2018 [ 5609.716491] Call Trace: [ 5609.717566] [&lt;ffffffffa02077dc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 5609.719016] [&lt;ffffffffa020788c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 5609.722670] [&lt;ffffffffa0fe5ab4&gt;] mdd_parent_fid+0x374/0x3b0 [mdd] [ 5609.724305] [&lt;ffffffffa0fe5bc0&gt;] mdd_is_parent+0xd0/0x1a0 [mdd] [ 5609.725641] [&lt;ffffffffa0fe5e94&gt;] mdd_is_subdir+0x204/0x240 [mdd] [ 5609.726669] [&lt;ffffffffa10642d0&gt;] mdt_reint_rename_internal.isra.46+0x810/0x2750 [mdt] [ 5609.728468] [&lt;ffffffffa106e14b&gt;] mdt_reint_rename_or_migrate.isra.51+0x19b/0x860 [mdt] [ 5609.730274] [&lt;ffffffffa106e843&gt;] mdt_reint_rename+0x13/0x20 [mdt] [ 5609.731149] [&lt;ffffffffa106e8d0&gt;] mdt_reint_rec+0x80/0x210 [mdt] [ 5609.732097] [&lt;ffffffffa104b723&gt;] mdt_reint_internal+0x6e3/0xab0 [mdt] [ 5609.732988] [&lt;ffffffffa10568e7&gt;] mdt_reint+0x67/0x140 [mdt] [ 5609.734283] [&lt;ffffffffa05f5605&gt;] tgt_request_handle+0xaf5/0x1590 [ptlrpc] [ 5609.735808] [&lt;ffffffffa05993a9&gt;] ptlrpc_server_handle_request+0x259/0xad0 [ptlrpc] [ 5609.737741] [&lt;ffffffffa059d36c&gt;] ptlrpc_main+0xb5c/0x2040 [ptlrpc] [ 5609.738705] [&lt;ffffffff810b4ed4&gt;] kthread+0xe4/0xf0 [ 5609.739563] [&lt;ffffffff817c4c77&gt;] ret_from_fork_nospec_end+0x0/0x39 [ 5609.740589] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [ 5609.741650] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35047" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35047</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> mdd: use mdd_object_fid() instead of mdo2fid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cfa0e83b83cca5b47c005d3934ee1abba07313ba</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35047/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35047/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11668" title="mdd_parent_fid()) ASSERTION( (((mdd_object_type(obj)) &amp; 00170000) == 0040000) ) failed" class="issue-link" data-issue-key="LU-11668"><del>LU-11668</del></a> mdd: use mdd_object_fid() instead of mdo2fid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7de9babe6f9af6dfdb20360211f8ecea344b0500</p> Related Development Rank 1|i006dz: Rank (Obsolete) 9223372036854775807