https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7453 Lustre <p>Wednesday morning one of our production MDS nodes hit an assertion:</p> <p>{{<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.384335&#93;</span> LustreError: 14428:0:(osp_sync.c:352:osp_sync_interpret()) ASSERTION( rc || req-&gt;rq_transno ) failed:<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.396346&#93;</span> LustreError: 14428:0:(osp_sync.c:352:osp_sync_interpret()) LBUG<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.404445&#93;</span> Pid: 14428, comm: ptlrpcd_2<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.409032&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.409033&#93;</span> Call Trace:<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.414039&#93;</span> <span class="error">&#91;&lt;ffffffffa0430895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.422141&#93;</span> <span class="error">&#91;&lt;ffffffffa0430e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.429363&#93;</span> <span class="error">&#91;&lt;ffffffffa0f6e3db&gt;&#93;</span> osp_sync_interpret+0x50b/0x510 <span class="error">&#91;osp&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.437003&#93;</span> <span class="error">&#91;&lt;ffffffffa075aacd&gt;&#93;</span> ptlrpc_check_set+0x31d/0x1c20 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.444806&#93;</span> <span class="error">&#91;&lt;ffffffff8108802b&gt;&#93;</span> ? try_to_del_timer_sync+0x7b/0xe0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.452147&#93;</span> <span class="error">&#91;&lt;ffffffffa0788b13&gt;&#93;</span> ptlrpcd_check+0x3d3/0x610 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.459582&#93;</span> <span class="error">&#91;&lt;ffffffffa078924b&gt;&#93;</span> ptlrpcd+0x20b/0x370 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.466413&#93;</span> <span class="error">&#91;&lt;ffffffff81064c00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.473654&#93;</span> <span class="error">&#91;&lt;ffffffffa0789040&gt;&#93;</span> ? ptlrpcd+0x0/0x370 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.480485&#93;</span> <span class="error">&#91;&lt;ffffffff8109e78e&gt;&#93;</span> kthread+0x9e/0xc0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.486243&#93;</span> <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.492100&#93;</span> <span class="error">&#91;&lt;ffffffff8109e6f0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.497954&#93;</span> <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.503992&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.506435&#93;</span> Kernel panic - not syncing: LBUG<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.511515&#93;</span> Pid: 14428, comm: ptlrpcd_2 Not tainted 2.6.32-504.30.3.el6.atlas.x86_64 #1<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.520881&#93;</span> Call Trace:<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.523916&#93;</span> <span class="error">&#91;&lt;ffffffff81529cbc&gt;&#93;</span> ? panic+0xa7/0x16f<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.529778&#93;</span> <span class="error">&#91;&lt;ffffffffa0430eeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.537188&#93;</span> <span class="error">&#91;&lt;ffffffffa0f6e3db&gt;&#93;</span> ? osp_sync_interpret+0x50b/0x510 <span class="error">&#91;osp&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.545012&#93;</span> <span class="error">&#91;&lt;ffffffffa075aacd&gt;&#93;</span> ? ptlrpc_check_set+0x31d/0x1c20 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.553002&#93;</span> <span class="error">&#91;&lt;ffffffff8108802b&gt;&#93;</span> ? try_to_del_timer_sync+0x7b/0xe0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.560340&#93;</span> <span class="error">&#91;&lt;ffffffffa0788b13&gt;&#93;</span> ? ptlrpcd_check+0x3d3/0x610 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.567961&#93;</span> <span class="error">&#91;&lt;ffffffffa078924b&gt;&#93;</span> ? ptlrpcd+0x20b/0x370 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.574979&#93;</span> <span class="error">&#91;&lt;ffffffff81064c00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.582209&#93;</span> <span class="error">&#91;&lt;ffffffffa0789040&gt;&#93;</span> ? ptlrpcd+0x0/0x370 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.589034&#93;</span> <span class="error">&#91;&lt;ffffffff8109e78e&gt;&#93;</span> ? kthread+0x9e/0xc0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.594981&#93;</span> <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.601024&#93;</span> <span class="error">&#91;&lt;ffffffff8109e6f0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> 2015-11-18 10:58:35 <span class="error">&#91;1912759.606874&#93;</span> <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> }}</p> <p>Is this related to <a href="https://jira.hpdd.intel.com/browse/LU-5629" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-5629</a> ?</p> 2.5.4-2.6.32_504.30.3.el6.atlas.x86_64.x86_64<br/> LU-7453

osp_sync_interpret assertion

Bug Major Resolved Cannot Reproduce Alex Zhuravlev Jesse Hanley Fri, 20 Nov 2015 13:08:19 +0000 Thu, 11 Feb 2016 18:08:44 +0000 Thu, 11 Feb 2016 18:08:44 +0000 Lustre 2.5.4 0 8 <p>Hi Dmitry,</p> <p>It looks like the assertion and back traces in this ticket are the same as those in <a href="https://jira.whamcloud.com/browse/LU-5629" title="osp_sync_interpret() ASSERTION( rc || req-&gt;rq_transno ) failed" class="issue-link" data-issue-key="LU-5629"><del>LU-5629</del></a>, which was not resolved by the fix of <a href="https://jira.whamcloud.com/browse/LU-3892" title="osp_sync.c:356:osp_sync_interpret()) ASSERTION( req-&gt;rq_transno == 0 ) failed" class="issue-link" data-issue-key="LU-3892"><del>LU-3892</del></a>.</p> <p>Could you please take a look at this ticket as well? Thank you.</p> <p>is it possible to get any logs for the case? the most valuable information at the moment is info on RPC (was it OST_DESTROY or OST_SETATTR).</p> <p>Hi Jesse,</p> <p>Does this assertion still occur? Is it possible to gather Lustre debug logs required by Alex above? Thank you.</p> <p>Hey Alex and Jian,</p> <p>Sorry for not responding sooner. We haven't seen this assertion since I submitted this. I don't have any debug logs from when it happened, but I do have a vmcore dump. I'll take a look and see if I can find anything relevant in it (not sure if it'll be there).</p> <p>Thank you very much, Jesse.</p> <p>From the backtrace, it looks like <tt>ptlrpcd</tt> called <tt>ptlrpcd_check</tt>, which called <tt>ptlrpc_check_set</tt>, which finally called <tt>osp_sync_interpret</tt>.</p> <p>If I followed the code correctly (very likely that I didn't - I'm not that familiar with the internals of Lustre), it looks like <tt>ptlrpc_req_interpret</tt> is only called from <tt>ptlrpc_set_destroy</tt> and <tt>ptlrpc_check_set</tt>. Am I right in assuming that <tt>ptlrpc_set_destroy</tt> corresponds to an OST_DESTROY and <tt>ptlrpc_check_set</tt> corresponds to a OST_SETATTR call? If so, does that mean this is associated with an OST_SETATTR call?</p> <p>Sorry if this isn't much help; I'm still learning how to dig around core dumps.</p> <p>Hi Alex,<br/> Could you please take a look at Jesse's comments and advise? Thank you.</p> <p>there is a ticket with a binary and instructions: <a href="https://bugzilla.lustre.org/show_bug.cgi?id=13155" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=13155</a><br/> if you've got time - please, try to extract the log from the dump. thanks in advance!</p> <p>I've uploaded the log. Sorry for the delay.</p> <p>I've never seen this extension for crash before. Thanks for the info!</p> <p>&#8211;<br/> Jesse</p> <p>Jesse -</p> <p>Just FYI, those ptlrpc functions aren't related to the higher level operations you're describing. They both act on sets of RPCs, so they don't have anything to do with SETATTR or DESTROY.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Hi Alex,</p> <p>Jesse has uploaded the logs. Could you please investigate and suggest? Thank you.</p> <p>it was a resend for OST_DESTROY, though I have no a good idea for the root cause yet..</p> <p>yes, we can add a bit more debug.. which target branch is supposed?</p> <p>Thank you, Alex.</p> <p>The server version is Lustre 2.5.4 (2.5.4-2.6.32_504.30.3.el6.atlas.x86_64.x86_64).</p> <p>any additional patches on top of that?</p> <p>As per ORNL ok to close - this just happened one time a long time ago</p> Related LU-5629 Development Rank 1|hzxttb: Rank (Obsolete) 9223372036854775807 Severity