https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5565 Lustre <p>I see this running racer with MDSCOUNT=4. I simplified it a bit.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># export MDSCOUNT=4 # export MOUNT_2=y # llmount.sh ... # cd /mnt/lustre # <span class="code-keyword">while</span> <span class="code-keyword">true</span>; <span class="code-keyword">do</span> lfs mkdir -c4 d0; sys_open d0/f0 cw; rm -rf d0; done &amp; # cd /mnt/lustre2 # <span class="code-keyword">while</span> <span class="code-keyword">true</span>; <span class="code-keyword">do</span> setfattr -n user.0 -v 0 d0; done </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>u login: [ 6081.349963] LustreError: 5860:0:(osd_handler.c:1959:osd_attr_set()) ASSERTION( dt_object_exists(dt) &amp;&amp; !dt_object_remote(dt) ) failed: [ 6081.354491] LustreError: 5860:0:(osd_handler.c:1959:osd_attr_set()) LBUG [ 6081.356572] Pid: 5860, comm: ll_ost_out01_00 [ 6081.357962] [ 6081.357964] Call Trace: [ 6081.359351] [&lt;ffffffffa02be8c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 6081.361546] [&lt;ffffffffa02beec7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 6081.363579] [&lt;ffffffffa0b2d351&gt;] osd_attr_set+0x1f1/0x540 [osd_ldiskfs] [ 6081.365686] [&lt;ffffffffa0b1f07f&gt;] ? osd_object_write_lock+0x9f/0x130 [osd_ldiskfs] [ 6081.367802] [&lt;ffffffffa06fb838&gt;] out_tx_attr_set_exec+0x208/0x370 [ptlrpc] [ 6081.369162] [&lt;ffffffffa06f758a&gt;] out_tx_end+0xda/0x5c0 [ptlrpc] [ 6081.370361] [&lt;ffffffffa06ff0b9&gt;] out_handle+0x5e9/0xdf0 [ptlrpc] [ 6081.371583] [&lt;ffffffffa069207c&gt;] ? lustre_msg_get_opc+0x9c/0x110 [ptlrpc] [ 6081.372926] [&lt;ffffffffa06f461e&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] [ 6081.374258] [&lt;ffffffffa06a4a67&gt;] ptlrpc_main+0xe27/0x1980 [ptlrpc] [ 6081.375509] [&lt;ffffffffa06a3c40&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] [ 6081.376702] [&lt;ffffffff8109eab6&gt;] kthread+0x96/0xa0 [ 6081.377643] [&lt;ffffffff8100c30a&gt;] child_rip+0xa/0x20 [ 6081.378592] [&lt;ffffffff81554710&gt;] ? _spin_unlock_irq+0x30/0x40 [ 6081.379716] [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [ 6081.380747] [&lt;ffffffff8109ea20&gt;] ? kthread+0x0/0xa0 [ 6081.381702] [&lt;ffffffff8100c300&gt;] ? child_rip+0x0/0x20 [ 6081.382681] [ 6081.383251] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Here are some stack traces from a different occurrence.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 9780 TASK: ffff8801d69708c0 CPU: 0 COMMAND: "setfattr" #0 [ffff8801d6bcf6f8] schedule at ffffffff81551457 #1 [ffff8801d6bcf7c0] schedule_timeout at ffffffff81552373 #2 [ffff8801d6bcf890] ptlrpc_set_wait at ffffffffa0688c4a [ptlrpc] #3 [ffff8801d6bcf930] ptlrpc_queue_wait at ffffffffa0689217 [ptlrpc] #4 [ffff8801d6bcf950] mdc_xattr_common at ffffffffa0912d53 [mdc] #5 [ffff8801d6bcf9e0] mdc_setxattr at ffffffffa0913146 [mdc] #6 [ffff8801d6bcfa30] lmv_setxattr at ffffffffa08ca50b [lmv] #7 [ffff8801d6bcfac0] ll_setxattr_common at ffffffffa0e746f0 [lustre] #8 [ffff8801d6bcfb80] ll_setxattr at ffffffffa0e75054 [lustre] #9 [ffff8801d6bcfd10] __vfs_setxattr_noperm at ffffffff811caeee #10 [ffff8801d6bcfd70] vfs_setxattr at ffffffff811cb0c4 #11 [ffff8801d6bcfdc0] setxattr at ffffffff811cb1a0 #12 [ffff8801d6bcff10] sys_setxattr at ffffffff811cb488 #13 [ffff8801d6bcff80] system_call_fastpath at ffffffff8100b072 RIP: 000000377fee53c9 RSP: 00007fff08cfd300 RFLAGS: 00010206 RAX: 00000000000000bc RBX: ffffffff8100b072 RCX: 00000000014f0100 RDX: 00000000014f0000 RSI: 00007fff08cff703 RDI: 00007fff08cff70f RBP: 00007fff08cff703 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000000 R14: 00007fff08cfd580 R15: 00000000014f0000 ORIG_RAX: 00000000000000bc CS: 0033 SS: 002b PID: 3693 TASK: ffff880202176500 CPU: 0 COMMAND: "mdt00_000" #0 [ffff880202179790] schedule at ffffffff81551457 #1 [ffff880202179858] schedule_timeout at ffffffff81552373 #2 [ffff880202179928] ptlrpc_set_wait at ffffffffa0688c4a [ptlrpc] #3 [ffff8802021799c8] ptlrpc_queue_wait at ffffffffa0689217 [ptlrpc] #4 [ffff8802021799e8] out_remote_sync at ffffffffa07001b1 [ptlrpc] #5 [ffff880202179a38] osp_trans_trigger at ffffffffa0d9e4aa [osp] #6 [ffff880202179a78] osp_trans_start at ffffffffa0d9f914 [osp] #7 [ffff880202179aa8] lod_trans_start at ffffffffa0d27c11 [lod] #8 [ffff880202179ae8] mdd_trans_start at ffffffffa0c1c067 [mdd] #9 [ffff880202179af8] mdd_unlink at ffffffffa0c0175a [mdd] #10 [ffff880202179bb8] mdo_unlink at ffffffffa0c6a688 [mdt] #11 [ffff880202179bc8] mdt_reint_unlink at ffffffffa0c74468 [mdt] #12 [ffff880202179c78] mdt_reint_rec at ffffffffa0c6a421 [mdt] #13 [ffff880202179c98] mdt_reint_internal at ffffffffa0c4fc63 [mdt] #14 [ffff880202179cd8] mdt_reint at ffffffffa0c504cb [mdt] #15 [ffff880202179d18] tgt_request_handle at ffffffffa06f461e [ptlrpc] #16 [ffff880202179d78] ptlrpc_main at ffffffffa06a4a67 [ptlrpc] #17 [ffff880202179eb8] kthread at ffffffff8109eab6 #18 [ffff880202179f48] kernel_thread at ffffffff8100c30a PID: 4642 TASK: ffff8801e59707c0 CPU: 1 COMMAND: "mdt00_004" #0 [ffff8801e856f7c0] schedule at ffffffff81551457 #1 [ffff8801e856f888] schedule_timeout at ffffffff81552373 #2 [ffff8801e856f958] ptlrpc_set_wait at ffffffffa0688c4a [ptlrpc] #3 [ffff8801e856f9f8] ptlrpc_queue_wait at ffffffffa0689217 [ptlrpc] #4 [ffff8801e856fa18] out_remote_sync at ffffffffa07001b1 [ptlrpc] #5 [ffff8801e856fa68] osp_trans_trigger at ffffffffa0d9e4aa [osp] #6 [ffff8801e856faa8] osp_trans_start at ffffffffa0d9f914 [osp] #7 [ffff8801e856fad8] lod_trans_start at ffffffffa0d27c11 [lod] #8 [ffff8801e856fb18] mdd_trans_start at ffffffffa0c1c067 [mdd] #9 [ffff8801e856fb28] mdd_attr_set at ffffffffa0c14985 [mdd] #10 [ffff8801e856fba8] mo_attr_set at ffffffffa0c74bc8 [mdt] #11 [ffff8801e856fbb8] mdt_reint_setxattr at ffffffffa0c75740 [mdt] #12 [ffff8801e856fc78] mdt_reint_rec at ffffffffa0c6a421 [mdt] #13 [ffff8801e856fc98] mdt_reint_internal at ffffffffa0c4fc63 [mdt] #14 [ffff8801e856fcd8] mdt_reint at ffffffffa0c504cb [mdt] #15 [ffff8801e856fd18] tgt_request_handle at ffffffffa06f461e [ptlrpc] #16 [ffff8801e856fd78] ptlrpc_main at ffffffffa06a4a67 [ptlrpc] #17 [ffff8801e856feb8] kthread at ffffffff8109eab6 #18 [ffff8801e856ff48] kernel_thread at ffffffff8100c30a PID: 4225 TASK: ffff8801ed6c87c0 CPU: 1 COMMAND: "ll_ost_out00_00" #0 [ffff8801ed6cb998] machine_kexec at ffffffff81039950 #1 [ffff8801ed6cb9f8] crash_kexec at ffffffff810d4372 #2 [ffff8801ed6cbac8] panic at ffffffff81550d83 #3 [ffff8801ed6cbb48] lbug_with_loc at ffffffffa02bef1b [libcfs] #4 [ffff8801ed6cbb68] osd_attr_set at ffffffffa0b2d351 [osd_ldiskfs] #5 [ffff8801ed6cbbc8] out_tx_attr_set_exec at ffffffffa06fb838 [ptlrpc] #6 [ffff8801ed6cbc08] out_tx_end at ffffffffa06f758a [ptlrpc] #7 [ffff8801ed6cbc58] out_handle at ffffffffa06ff0b9 [ptlrpc] #8 [ffff8801ed6cbd18] tgt_request_handle at ffffffffa06f461e [ptlrpc] #9 [ffff8801ed6cbd78] ptlrpc_main at ffffffffa06a4a67 [ptlrpc] #10 [ffff8801ed6cbeb8] kthread at ffffffff8109eab6 #11 [ffff8801ed6cbf48] kernel_thread at ffffffff8100c30a </pre> </div></div> <p>I'm not sure why the handler is ll_ost_out00_00 but I checked and that task is handling a OUT_UPDATE RPC from lustre-MDT0000 to lustre-MDT0002.</p> <p>Note that the setattr is for the ctime on d0. Should this ctime update really go to each stripe? If so then shouldn't we lock the stripes as we do from mdt_attr_set()?</p> LU-5565

(osd_handler.c:1959:osd_attr_set()) ASSERTION( dt_object_exists(dt) && !dt_object_remote(dt) ) failed

Bug Blocker Resolved Fixed John Hammond John Hammond MB dne2 mdt out Fri, 29 Aug 2014 17:53:30 +0000 Wed, 3 Feb 2016 19:19:56 +0000 Wed, 3 Feb 2016 19:19:56 +0000 Lustre 2.7.0 Lustre 2.7.0 0 10 <p>I hit this same LASSERT running racer on my single-node test system (2x1GB MDT, 3x4GB OST) as part of acceptance-small.sh using v2_6_54_0-16-g0024956:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 1492:0:(osd_handler.c:1963:osd_attr_set()) ASSERTION( dt_object_exists(dt) &amp;&amp; !dt_object_remote(dt) ) failed: Pid: 1492, comm: ll_ost_out00_00 libcfs_debug_dumpstack+0x55/0x80 [libcfs] lbug_with_loc+0x47/0xb0 [libcfs] osd_attr_set+0x197/0x4e0 [osd_ldiskfs] out_tx_attr_set_exec+0x260/0x3f0 [ptlrpc] out_tx_end+0xda/0x5c0 [ptlrpc] out_handle+0x7c0/0xe50 [ptlrpc] tgt_request_handle+0x71e/0xb10 [ptlrpc] ptlrpc_main+0xe64/0x1990 [ptlrpc] kthread+0x96/0xa0 </pre> </div></div> <p>I've pushed a cleanup/debug patch <a href="http://review.whamcloud.com/12398" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12398</a> to at least split this LASSERT() into two separate ones, so that it is easier to see which one is failing. This is not expected to fix the problem here.</p> <p>dt_object_exists() is failing.</p> <p>I hit this problem again, with the same stack in out_handle() as my previous occurrence.</p> <p>Can we replace this with error handling?</p> <p>yes, in out_tx_attr_set_exec() once the lock is taken we should check whether the object still exists.</p> <p>Please see <a href="http://review.whamcloud.com/12608" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12608</a>.</p> <p>"Note that the setattr is for the ctime on d0. Should this ctime update really go to each stripe? If so then shouldn't we lock the stripes as we do from mdt_attr_set()?"</p> <p>John, I think you are right, we should lock the object both for setting xattr and attr. And ctime does not need to go every stripe, but</p> <p>1. lod_attr_set needs to be fixed, i.e. set_attr time only goes to master object. <br/> 2. lmv_merge_attr needs to be fixed, i.e. it needs to take count in master object time as well, right now it only merge the time attributes from all of the sub-stripes. </p> <p>Can we replace this with error handling?</p> <p>Please do it in OSD, then local object and remote object are getting equal treatment.</p> <p>The real reason of this LBUG is because of the race between close(unlink orphan) and setxattr(or setattr). Because close is not protected by ldlm lock. In current implementation(mdd_close()), we use a local lock(mdd_write_lock) to protect the unlink orphan process in the MDD layer, which is fine if the directory is a local directory. But if the directory is a striped directory, the local lock might not be enough, because the unlink orphan (triggered by close) will delete the stripe on the other MDT as well. If other threads do setxattr(or setattr) at the same time on this striped directory. it causes this LBUG.<br/> There are a few options to fix this problem,</p> <p>1. protect close process with ldlm lock. <br/> 2. "Open" the file before setxattr (or setattr), of course only on the server side in MDD layer, then "close" the file after setxattr (or setattr), which might trigger unlink orphan as well.</p> <p>Either of them might need some changes (not tiny), probably not a good choice for 2.7. Any other suggestions?</p> <p>Isn't there a (already broken) rule against taking LDLM locks in close. From mdc_close()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* To avoid a livelock (bug 7034), we need to send CLOSE RPCs to a * portal whose threads are not taking any DLM locks and are therefore * always progressing */ req-&gt;rq_request_portal = MDS_READPAGE_PORTAL; ptlrpc_at_set_req_timeout(req); </pre> </div></div> <p>Why not fix this by handling the race?</p> <p>using LDLM lock has additional benefits as an object can be accessed by another node and LDLM is the natural mechanism to control access.</p> <p>Couldn't we just return -ENOENT in this case? The setxattr() call is path-based, and not file descriptor based, so there is no expectation that it is free of races. If the object is deleted then it is perfectly fine to return an error to the caller. </p> <p>I had thought the same thing, then I saw <a href="https://jira.whamcloud.com/browse/LU-6027" title="Issues with EAs of orphan files and EAs with empty values" class="issue-link" data-issue-key="LU-6027"><del>LU-6027</del></a>, and discussed with Li Wei. The problem is if the application open the file and hold the file handle, do setxattr, should we expect succeeds or failure? And it does succeed on ext4? I do not know if posix require this or not? And also if we return -ENOENT here, Li Wei needs to change his new "racer" program and disable sanityn.sh 82.</p> <p>&gt; I had thought the same thing, then I saw <a href="https://jira.whamcloud.com/browse/LU-6027" title="Issues with EAs of orphan files and EAs with empty values" class="issue-link" data-issue-key="LU-6027"><del>LU-6027</del></a>, and discussed with Li Wei. The problem is if the application open the file and hold the file handle, do setxattr, should we expect succeeds or failure? And it does succeed on ext4? I do not know if posix require this or not? And also if we return -ENOENT here, Li Wei needs to change his new "racer" program and disable sanityn.sh 82.</p> <p>setxattr() or fsetxattr()?</p> <p>I meant Li Wei used this file handle thing to set xattr in his new "racer" test program. I am not sure which one he used. But if I returned ENOENT here, it will definitely cause sanityn.sh 82 failed. </p> <p>We should be clear about that before we add LDLM locks to close because of it.</p> <p>Note that <a href="http://review.whamcloud.com/#/c/12608/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12608/</a> does <b>not</b> cause -ENOENT to be returned under the following situation.</p> <ol> <li>lfs mkdir -c4 d0</li> <li>fd = open("d0", O_RDONLY)</li> <li>rmdir("d0")</li> <li>fsetxattr(fd, "user.foo", "bar", 3, 0)</li> </ol> <p>Yes, right now, it does not return ENOENT, because of the file is still open, and we allow this in current implementation. So is it bad we just return ENOENT for fsetxattr in this case? </p> <p>&gt; Yes, right now, it does not return ENOENT, because of the file is still open, and we allow this in current implementation. So is it bad we just return ENOENT for fsetxattr in this case?</p> <p>Why would we do that?</p> <p>I mean If it return ENOENT for unlinked object during setattr or setxattr, then we do not need worry about orphans. </p> <p>Why not use <a href="http://review.whamcloud.com/#/c/12608/?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12608/?</a></p> <p>It is different problem. I thought we are talking about about resolving the race, and <a href="http://review.whamcloud.com/#/c/12608" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12608</a> is to avoid panic, which actually is covering the problem somehow. IMHO</p> <p>It is the same problem. <a href="http://review.whamcloud.com/#/c/12608" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12608</a> handles the race.</p> <p>Hmm, I thought we suppose to use ldlm lock to protect the race like this, or just not get into setattr or set xattr, once the object is being unlinked. </p> <p>And also I thought the object should exist once the modification getting into OSD/OSP layer, because it suppose to do sanity check and hold ldlm lock in MDT layer. Though there are some exception for now, like close. I thought we should resolve this in MDT layer, instead of "covering" it in OSD/OSP. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12608/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12608/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5565" title="(osd_handler.c:1959:osd_attr_set()) ASSERTION( dt_object_exists(dt) &amp;&amp; !dt_object_remote(dt) ) failed" class="issue-link" data-issue-key="LU-5565"><del>LU-5565</del></a> osd-ldiskfs: handle non-existing objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 30b0b011dfd53faf8f903401fa0539a8d625b0e5</p> <p>Note that osd-zfs still needs to be fixed.</p> <p>Patch landed to Master.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13496" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13496</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5565" title="(osd_handler.c:1959:osd_attr_set()) ASSERTION( dt_object_exists(dt) &amp;&amp; !dt_object_remote(dt) ) failed" class="issue-link" data-issue-key="LU-5565"><del>LU-5565</del></a> osd-ldiskfs: handle nonexisting objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0b4fd760b34b1ded5e8fee6db07a5a0f0291389d</p> <p>We hit the same trouble with ZFS backend on lola. We need enhance the patch <a href="http://review.whamcloud.com/#/c/12608/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12608/</a> for ZFS case.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jan 27 05:49:12 lola-5 kernel: LustreError: 22690:0:(osd_object.c:925:osd_attr_set()) ASSERTION( dt_object_exists(dt) ) failed: Jan 27 05:49:12 lola-5 kernel: LustreError: 22690:0:(osd_object.c:925:osd_attr_set()) LBUG </pre> </div></div> <p>this is done in <a href="http://review.whamcloud.com/#/c/18024/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18024/</a> and <a href="http://review.whamcloud.com/#/c/18155/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18155/</a></p> <p>For build '20160201' (master) executed during soaktest (see: <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160201" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160201</a>)<br/> the error happened again and persists even after reboot of the node so that the affected <br/> OSS node and OSTs become unusable.<br/> The MDTs have been formated with <em>ldiskfs</em>, OSTs with <em>zfs</em></p> <p>Sequence of events:</p> <ul> <li>Feb 2 01:32:18 2016: lfsck started on mds-0 (<tt>lola-8</tt> using the command {{lctl lfsck_start -M soaked-MDT0000 -A }}<br/> OSS nodes <tt>lola-2</tt> crashed with LBUG (See see error message after time stamp in console log 'Feb 2 01:30:01')</li> <li>Feb 2 01:50:01 2016: lola-2 rebooted; remounted OSTs node crashed with same LBUG<br/> (see error message after 'Feb 2 01:50:01 lola-2 TIME: Time stamp for console' in console log)</li> <li>Several tries to simply reboot the node and remount again</li> <li>Feb 2 02:13:35 PST 2016: Execution of <tt>lctl lfsck_stop -M soaked-MDT0000 -A }} end successfull. No OSTs mounted on {{lola-2</tt></li> <li>Feb 2 02:15:01 2016: Mounted OSTs on <tt>lola-2</tt>. Node crashed almost immediately with same LBUG again. Forced creation of debug log with 'lfsck' enable and buffer size 512M<br/> (see attached file: lustre-log-lu-5565)</li> <li>Feb 2 02:31 2016: Rebooted and re-mounted OSTs one more with 'panic_on_lbug', lfsck enabled, debug buffer 512M<br/> Explicitly forced stack trace.<br/> Also debug logs were written after LBUG appear again (almost immediately). See:<br/> lustre-log.1454409077.7931<br/> lustre-log.1454409090.7841</li> <li>OSS node can't be used as FS resource anymore</li> </ul> <p>Attached files:</p> <ul> <li><tt>lola-2</tt> message and console log</li> <li>debug-log after <tt>lfsck</tt> stopped and remount of OSTs</li> <li>lustre-log.1454409077.7931, lustre-log.1454409090.7841 - Debug logs of server in current status<br/> when mounting the OSTs</li> </ul> <p>Please open a new ticket to track any further work that is needed in this area for 2.8</p> Duplicate LU-6100 Related LU-2821 Development Rank 1|hzwuzb: Rank (Obsolete) 15517 Severity