https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6664 Lustre <p>On all of our filesystems, the following error message is extremely common:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 8746:0:(ost_handler.c:1776:ost_blocking_ast()) Error -2 syncing data on lock cancel </pre> </div></div> <p>There is nothing else in the logs that gives any hint as to why this message is appearing.</p> <p>Our filesystems all use osd-zfs, and we are currently running Lustre 2.5.3-5chaos (see github.com/chaos/lustre).</p> <p>If this is a symptom of a bug, then please fix it. If this is not a symptom of a bug, then please stop scaring our system administrators with this message.</p> LU-6664

(ost_handler.c:1765:ost_blocking_ast()) Error -2 syncing data on lock cancel

Bug Minor Resolved Fixed Zhenyu Xu Christopher Morrone llnl Fri, 29 May 2015 19:20:16 +0000 Wed, 27 Jul 2016 18:54:19 +0000 Mon, 31 Aug 2015 21:14:46 +0000 Lustre 2.5.3 0 10 <p>Bobijam</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Bobi: This message is ENOENT when we are supposedly trying to flush data on lock cancel. But technically if we have a lock, the object should be there (the only exception I can think of is the actual object destroy is happening under the lock, so at the end of destroy the lock is still there and the object is not, but then there should be nothing to flush).<br/> So can you please examine server side code for lock cancel to see if there are any possible races that could lead to this message.</p> <p>Chris,</p> <p>Does your system just undergo recovery before these messages appears?</p> <p>No, it has not just undergone recovery. There are no other messages in the logs surrounding these ost_blocking_ast() messages.</p> <p><a href="http://review.whamcloud.com/15167" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15167</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre>LU-6664 ofd: LDLM lock should cover object destroy The exclusive PW lock protecting OST object destroy should be released after object destroy procedure. Quench error messages of object unavailability when trying to cancel a LDLM lock. </pre> </div></div> <p>Just as I was reviewing this patch (the comments are n the patch), I just remembered that LLNL had suspicions of double referenced objects in the past (<a href="https://jira.whamcloud.com/browse/LU-5648" title="corrupt files contain extra data" class="issue-link" data-issue-key="LU-5648"><del>LU-5648</del></a>) where same object was potentially referenced twice (that was never confirmed, though).<br/> So having objects owned by two files would most likely lead to this message too.</p> <p>I wonder if lfsck in 2.5.4 is already in a good enough shape to be able to detect that.</p> <p>The LFSCK in 2.5.x does not check MDT&lt;-&gt;OST consistency. That feature ("lctl lfsck_start -t layout") wasn't added until 2.6.0.</p> <p>Bobijam, can you please also make a version of your patch for master.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/15997" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15997</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6664" title="(ost_handler.c:1765:ost_blocking_ast()) Error -2 syncing data on lock cancel" class="issue-link" data-issue-key="LU-6664"><del>LU-6664</del></a> ofd: LDLM lock should cover object destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4575d04887bfd2a78a5a0340841d2da6ef23c165</p> <p>Oleg and I looked into this issue more closely, and the current patch doesn't really solve the problem, since the race is when the two destroy threads are getting and dropping the DLM lock, and not when the actual destroy is happening. In master, the equivalent function <tt>tgt_blocking_ast()</tt> already has a check for <tt>dt_object_exists()</tt> and skips the call into <tt>ofd_sync()</tt> that generates this message completely.</p> <p>I think the right fix (for 2.5.x only) is to just skip this message for <tt>rc == -ENOENT</tt> as is already done in master.</p> <p>I too see ton's of these error messages: Any help in resolving them will be very helpful. I can provide any debug logs if required. It is consistently appearing in most of the OSS's. </p> <p>LustreError: 7577:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel<br/> LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel<br/> LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel<br/> LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 1 previous similar message<br/> LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel<br/> LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 2 previous similar messages<br/> LustreError: 33552:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel</p> <p>Thank you,<br/> Amit</p> <p>As per LLNL ok to close</p> Duplicate LU-7007 Related LU-5805 LU-7308 Development Rank 1|hzxein: Rank (Obsolete) 9223372036854775807 Severity