https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7579 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># export MDSCOUNT=4 # llmount.sh ... # cd /mnt/lustre # mkdir d0 # exec 7&lt;d0 # cd d0 # rmdir ../d0 # lfs mkdir -i1 d1 # exec 7&gt;&amp;- </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 130.331201] LustreError: 4033:0:(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed: [ 130.335818] LustreError: 4033:0:(osd_handler.c:2683:osd_object_destroy()) LBUG [ 130.338084] Pid: 4033, comm: mdt_rdpg00_001 [ 130.339427] [ 130.339429] Call Trace: [ 130.340776] [&lt;ffffffffa09338b5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 130.342927] [&lt;ffffffffa0933eb7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 130.344861] [&lt;ffffffffa12aa9b1&gt;] osd_object_destroy+0x5a1/0x5b0 [osd_ldiskfs] [ 130.346272] [&lt;ffffffffa144b49d&gt;] lod_sub_object_destroy+0x1fd/0x440 [lod] [ 130.347589] [&lt;ffffffffa1440080&gt;] lod_object_destroy+0x130/0x770 [lod] [ 130.348877] [&lt;ffffffffa093f751&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 130.350151] [&lt;ffffffffa1336ca8&gt;] mdd_close+0xa48/0xbf0 [mdd] [ 130.351286] [&lt;ffffffffa139e1c9&gt;] mdt_mfd_close+0x359/0x1980 [mdt] [ 130.352523] [&lt;ffffffffa0aa4e3d&gt;] ? class_handle_unhash_nolock+0x2d/0x150 [obdclass] [ 130.354034] [&lt;ffffffffa139fa04&gt;] mdt_close_internal+0x214/0x4f0 [mdt] [ 130.355298] [&lt;ffffffffa139ff9a&gt;] mdt_close+0x2ba/0x900 [mdt] [ 130.356495] [&lt;ffffffffa0d538f7&gt;] tgt_request_handle+0x907/0x14a0 [ptlrpc] [ 130.357874] [&lt;ffffffffa0cfdd7a&gt;] ptlrpc_main+0xdaa/0x18b0 [ptlrpc] [ 130.359089] [&lt;ffffffff8105e59d&gt;] ? finish_task_switch+0x7d/0x110 [ 130.360256] [&lt;ffffffff8105e568&gt;] ? finish_task_switch+0x48/0x110 [ 130.361445] [&lt;ffffffff81553065&gt;] ? thread_return+0x4e/0x7e9 [ 130.362538] [&lt;ffffffff810b6b6d&gt;] ? lock_release_holdtime+0x3d/0x190 [ 130.363791] [&lt;ffffffffa0cfcfd0&gt;] ? ptlrpc_main+0x0/0x18b0 [ptlrpc] [ 130.365028] [&lt;ffffffff8109e856&gt;] kthread+0x96/0xa0 [ 130.365935] [&lt;ffffffff8100c30a&gt;] child_rip+0xa/0x20 [ 130.366896] [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [ 130.367938] [&lt;ffffffff8109e7c0&gt;] ? kthread+0x0/0xa0 [ 130.368911] [&lt;ffffffff8100c300&gt;] ? child_rip+0x0/0x20 [ 130.369926] </pre> </div></div> LU-7579

(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode->i_nlink == 1 || inode->i_nlink == 2 ) failed

Bug Blocker Resolved Fixed Alex Zhuravlev John Hammond Thu, 17 Dec 2015 21:07:15 +0000 Wed, 28 Dec 2022 20:39:36 +0000 Sat, 13 Feb 2016 05:41:24 +0000 Lustre 2.8.0 Lustre 2.8.0 0 8 <p>Hmm, this is an interesting question. Once the parent(on MDT0) becomes orphan, we should not be able to create any file or directory.<br/> But if it create remote directory, then create RPC will be sent to another MDT(MDT1), in mdd_create(), MDT1 will retrieve the parent attributes from the MDT0 remotely, and since the parent is just an orphan linked to the orphan directory, so get_attr() have no way to tell the parent has become an orphan, so the creation go ahead, and failure happens.</p> <p>So we need permanently record this orphan flag somewhere in the inode, another flag inside ldiskfs or add another EA, either way this will impact the performance.</p> <p>how about store the orphan status in LMA, then get/set it through osd_attr_xxx()? other suggestion?</p> <p>Di, regarding your question in <a href="http://review.whamcloud.com/10274" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10274</a> </p> <blockquote> <p>Is it easy to add more flags here? Since I need a flag to indicate the directory is orphan, so no one can create sub-directory or file under this orphan anymore. </p> <p>right now we store this flag in memory of mdd object, but then it would not be transferred to remote target. (see <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a>)<br/> So the orphan has to be marked as a permanent flag in OSD layer, then osp_attr_get() can retrieve this flag, and aware of the orphan status. Please add comments in <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a>.</p></blockquote> <p>These flags are direct mapping a from the EXT4_*_FL inode flags, so it is a bit tricky to add new values here, since most of them are checked by e2fsck. </p> <p>There is one flag - <tt>EXT4_IMAGIC_FL</tt> which means that the inode is disconnected and does not have a parent, but it also needs a superblock feature flag to be set to be valid. </p> <p>It is also possible to prevent new files to be created in a directory by setting the nlink count = 0, I think, and adding it to add the orphan inode list. However, that is pretty ldiskfs-specific. </p> <p>What about a "compat" flag in the LMA? Is that sent to the remote MDT? That would at least be consistent with ZFS and ldiskfs. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>What about a "compat" flag in the LMA? Is that sent to the remote MDT? That would at least be consistent with ZFS and ldiskfs. </pre> </div></div> <p>That is what I thought as I well, but I am not sure if we should do this inside osd_attr_get/set(), i.e. get/set extra flags from LMA in osd_attr_get/set(), and input/output with lu_flags, or explicitly though get_xattr to retrieve these flags, which will definitely need extra RPC.</p> <p>it shouldn't be a problem to store additional EA - the directories usually have quite amount of free space in inode (given they don't store LOV). as for RPC RTT - we do have a notion of readahead for attributes/EAs/etc (already used by LFSCK), which can be employed by MDD to save RPC.<br/> in general - it's the same issue as opencounter. moving all these flags from mdd objects to an external storage is the right way to go.<br/> btw, what about LUSTRE_SLAVE_DEAD_FL ? we've been doing this for the slaves already?</p> <p>Yes, <tt>LUSTRE_SLAVE_DEAD_FL</tt> seems like exactly the right thing?</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17715" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17715</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a> mdd: Add ORPHAN/DEAD flag in LMA<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b8472fb5383e5a4ac477cbd2aaa16bb871660ffd</p> <ul class="alternate" type="square"> <li>umount of lustre on server resulted in similar ASSERTION</li> <li>env <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DNE setup, interop 2.7.64 server &lt;-&gt; 2.5.x client </pre> </div></div></li> <li>the dump <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-12-24 04:29:23 [592212.084691] LustreError: 76535:0:(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || in ode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed: 2015-12-24 04:29:23 [592212.100075] LustreError: 76535:0:(osd_handler.c:2683:osd_object_destroy()) LBUG 2015-12-24 04:29:23 [592212.107635] Pid: 76535, comm: mdt_rdpg00_001 2015-12-24 04:29:23 [592212.112078] 2015-12-24 04:29:23 [592212.112078] Call Trace: 2015-12-24 04:29:23 [592212.116383] [&lt;ffffffffa05ab875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2015-12-24 04:29:23 [592212.123536] [&lt;ffffffffa05abe77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2015-12-24 04:29:23 [592212.129909] [&lt;ffffffffa10d5fc1&gt;] osd_object_destroy+0x5a1/0x5b0 [osd_ldiskfs] 2015-12-24 04:29:23 [592212.137395] [&lt;ffffffffa118e05d&gt;] lod_sub_object_destroy+0x1fd/0x440 [lod] 2015-12-24 04:29:23 [592212.144466] [&lt;ffffffffa1182c70&gt;] lod_object_destroy+0x130/0x770 [lod] 2015-12-24 04:29:23 [592212.151180] [&lt;ffffffffa05b76c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2015-12-24 04:29:23 [592212.157999] [&lt;ffffffffa0f22e68&gt;] mdd_close+0xa48/0xbf0 [mdd] 2015-12-24 04:29:23 [592212.163938] [&lt;ffffffffa0f9d319&gt;] mdt_mfd_close+0x359/0x1980 [mdt] 2015-12-24 04:29:23 [592212.170354] [&lt;ffffffffa095309c&gt;] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] 2015-12-24 04:29:23 [592212.177782] [&lt;ffffffffa09532db&gt;] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] 2015-12-24 04:29:24 [592212.185218] [&lt;ffffffffa0952085&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2015-12-24 04:29:24 [592212.191881] [&lt;ffffffffa0709295&gt;] ? class_handle2object+0x95/0x190 [obdclass] 2015-12-24 04:29:24 [592212.199278] [&lt;ffffffffa0f9eb3a&gt;] mdt_close_internal+0x1fa/0x4e0 [mdt] 2015-12-24 04:29:24 [592212.206006] [&lt;ffffffffa0f9f0d7&gt;] mdt_close+0x2b7/0xa40 [mdt] 2015-12-24 04:29:24 [592212.211985] [&lt;ffffffffa09bdd4c&gt;] tgt_request_handle+0x8ec/0x1470 [ptlrpc] 2015-12-24 04:29:24 [592212.219081] [&lt;ffffffffa0964e71&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] 2015-12-24 04:29:24 [592212.225573] [&lt;ffffffffa0964030&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 2015-12-24 04:29:24 [592212.232020] [&lt;ffffffff8109ac66&gt;] kthread+0x96/0xa0 2015-12-24 04:29:24 [592212.237124] [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 2015-12-24 04:29:24 [592212.242272] [&lt;ffffffff8109abd0&gt;] ? kthread+0x0/0xa0 2015-12-24 04:29:24 [592212.247410] [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 2015-12-24 04:29:24 [592212.252722] 2015-12-24 04:29:24 [592212.254723] Kernel panic - not syncing: LBUG </pre> </div></div></li> <li>I will upload the necessary logs(lctl/dmesg).</li> </ul> <p><a href="http://review.whamcloud.com/#/c/18024/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18024/</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18024/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18024/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a> osd: move ORPHAN/DEAD flag to OSD<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 098fb363c3902f67b29ddfa864b452d0a8460ad2</p> <p>Landed for 2.8</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18444" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18444</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a> mdd: do not mark object as an orphan early<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a0d2710b49eac003f3777975a17a651d0996b5b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18444/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18444/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a> mdd: do not mark object as an orphan early<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1fd624857218bd5f7ae1529d3fee8933c9cb8a75</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49524" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49524</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7579" title="(osd_handler.c:2683:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode-&gt;i_nlink == 1 || inode-&gt;i_nlink == 2 ) failed" class="issue-link" data-issue-key="LU-7579"><del>LU-7579</del></a> osd-zfs: set orphan flag on the cached object<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: ce3e893bae93da83863d08e14f85bbdc205499f7</p> Duplicate Development End date Mon, 28 Dec 2015 21:07:15 +0000 Rank 1|hzxw7j: Rank (Obsolete) 9223372036854775807 Severity Start date Thu, 17 Dec 2015 21:07:15 +0000