https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3067 Lustre <p>One of our OSSs had problems writing to disk (due to a raid card problem). </p> <p>Several clients have an LBUG and haven't recovered after OSS reboot.<br/> The error is:</p> <p>Mar 29 06:20:10 cn492 kernel: LustreError: 3004:0:(osc_request.c:2357:brw_interpret()) ASSERTION(!(aa-&gt;aa_oa-&gt;o_valid &amp; OBD_MD_FLHANDLE)) failed<br/> Mar 29 06:20:10 cn492 kernel: LustreError: 3004:0:(osc_request.c:2357:brw_interpret()) LBUG</p> <p>I attach the associated log file, and reproduce some lines of context in /var/log/messages</p> <p>Mar 29 05:57:03 cn492 kernel: Lustre: lustre_0-OST0027-osc-ffff81021c041800: Connection restored to service lustre_0-OST0027 using nid 10.1.4.12<br/> 0@tcp.<br/> Mar 29 05:57:03 cn492 kernel: Lustre: Skipped 1 previous similar message<br/> Mar 29 06:09:39 cn492 kernel: Lustre: 3004:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1430341259304767 sent from lustre_0-OST002<br/> 7-osc-ffff81021c041800 to NID 10.1.4.120@tcp 756s ago has timed out (756s prior to deadline).<br/> Mar 29 06:09:39 cn492 kernel: req@ffff8101145e6800 x1430341259304767/t0 o3-&gt;lustre_0-OST0027_UUID@10.1.4.120@tcp:6/4 lens 448/592 e 1 to 1 dl <br/> 1364537379 ref 2 fl Rpc:/2/0 rc 0/0<br/> Mar 29 06:09:39 cn492 kernel: Lustre: 3004:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br/> Mar 29 06:09:39 cn492 kernel: Lustre: lustre_0-OST0027-osc-ffff81021c041800: Connection to service lustre_0-OST0027 via nid 10.1.4.120@tcp was l<br/> ost; in progress operations using this service will wait for recovery to complete.<br/> Mar 29 06:09:39 cn492 kernel: Lustre: Skipped 1 previous similar message<br/> Mar 29 06:09:39 cn492 kernel: Lustre: lustre_0-OST0027-osc-ffff81021c041800: Connection restored to service lustre_0-OST0027 using nid 10.1.4.12<br/> 0@tcp.<br/> Mar 29 06:09:39 cn492 kernel: Lustre: Skipped 1 previous similar message<br/> Mar 29 06:20:10 cn492 kernel: LustreError: 3004:0:(osc_request.c:2357:brw_interpret()) ASSERTION(!(aa-&gt;aa_oa-&gt;o_valid &amp; OBD_MD_FLHANDLE)) failed<br/> Mar 29 06:20:10 cn492 kernel: LustreError: 3004:0:(osc_request.c:2357:brw_interpret()) LBUG<br/> Mar 29 06:20:10 cn492 kernel: Pid: 3004, comm: ptlrpcd<br/> Mar 29 06:20:10 cn492 kernel: <br/> Mar 29 06:20:10 cn492 kernel: Call Trace:<br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff885786a1&gt;&#93;</span> libcfs_debug_dumpstack+0x51/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff88578bda&gt;&#93;</span> lbug_with_loc+0x7a/0xd0 <span class="error">&#91;libcfs&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff88580fc0&gt;&#93;</span> tracefile_init+0x0/0x110 <span class="error">&#91;libcfs&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8879c7e8&gt;&#93;</span> brw_interpret+0x8e8/0xdb0 <span class="error">&#91;osc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff886d36ac&gt;&#93;</span> after_reply+0xcac/0xe30 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff886d4b0b&gt;&#93;</span> ptlrpc_check_set+0x12db/0x15a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8004b396&gt;&#93;</span> try_to_del_timer_sync+0x7f/0x88<br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff887095ad&gt;&#93;</span> ptlrpcd_check+0xdd/0x1f0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8009a98c&gt;&#93;</span> process_timeout+0x0/0x5<br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff88709ef1&gt;&#93;</span> ptlrpcd+0x1b1/0x259 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8008f3ad&gt;&#93;</span> default_wake_function+0x0/0xe<br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8005dfc1&gt;&#93;</span> child_rip+0xa/0x11<br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff88709d40&gt;&#93;</span> ptlrpcd+0x0/0x259 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 29 06:20:10 cn492 kernel: <span class="error">&#91;&lt;ffffffff8005dfb7&gt;&#93;</span> child_rip+0x0/0x11<br/> Mar 29 06:20:10 cn492 kernel: <br/> Mar 29 06:20:10 cn492 kernel: LustreError: dumping log to /tmp/lustre-log.1364538010.3004</p> Scientific Linux [<a href='mailto:walker@fe02'>walker@fe02</a> ~]$ uname -r<br/> 2.6.18-348.3.1.el5<br/> <br/> Patchless client:<br/> <br/> lustre-client-modules-1.8.9-wc1_2.6.18_348.3.1.el5<br/> lustre-client-1.8.9-wc1_2.6.18_348.3.1.el5<br/> <br/> Servers are all running:<br/> [<a href='mailto:root@sn20'>root@sn20</a> ~]# rpm -qa | grep ^lustre<br/> lustre-modules-1.8.9-wc1_2.6.18_348.1.1.el5_lustre<br/> lustre-1.8.9-wc1_2.6.18_348.1.1.el5_lustre<br/> lustre-ldiskfs-3.1.53-wc1_2.6.18_348.1.1.el5_lustre<br/> <br/> LU-3067

ASSERTION(!(aa->aa_oa->o_valid & OBD_MD_FLHANDLE))

Bug Major Resolved Won't Fix Hongchao Zhang Christopher J. Walker mn8 Fri, 29 Mar 2013 11:29:21 +0000 Sat, 2 Jul 2016 15:09:38 +0000 Sat, 2 Jul 2016 15:09:38 +0000 Lustre 1.8.9 0 14 <p>We have seen this problem with our customer and <br/> while investigating it I stumbled upon the recent changes made to this code.<br/> <a href="http://jira.whamcloud.com/browse/LU-2703" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-2703</a><br/> This soft lockup issue was fixed in 1.8.9 by removing obd_cancel() (which cancels a ldlm lock held for brw) out of *_ap_completion. This was done as obd_cancel() is heavy operation and should not be called while holding a spinlock (cl_loi_list_lock). </p> <p>However this fix added the LASSERT,</p> <p> list_for_each_entry_safe(oap, tmp, &amp;aa-&gt;aa_oaps, oap_rpc_item) {<br/> list_del_init(&amp;oap-&gt;oap_rpc_item);</p> <p> aa-&gt;aa_oa-&gt;o_flags &amp;= ~OBD_FL_HAVE_LOCK;<br/> osc_ap_completion(cli, aa-&gt;aa_oa, oap, 1, rc);<br/> if (aa-&gt;aa_oa-&gt;o_flags &amp; OBD_FL_HAVE_LOCK) </p> { LASSERT(!(aa-&gt;aa_oa-&gt;o_valid &amp; OBD_MD_FLHANDLE)); LASSERT(index &lt; aa-&gt;aa_handle_count); aa-&gt;aa_handle[index++] = aa-&gt;aa_oa-&gt;o_handle; } <p>Shouldn't the OBD_MD_FLHANDLE be set as it indicates presence of valid lock handle ?</p> <p>FYI the customer is Sanger. </p> <p>The problem also occurs when the the network connection to an OSS fails (eg today when a colleague managed to take out both of the resilient core switches). </p> <p>Hongchao</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Girish,<br/> Thanks for your investigation! the OBD_MD_FLHANDLE could be used in client side (OSC)!</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,5971" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5971</a></p> <p>Hi I was wondering what the status of this issue was. We are running into this bug regularly at NOAA as well. Thanks.</p> <p>We ran into the same issue on 1.8.9 at FNAL on client. Trace dump is the same. It happened after client communication error with OSS (the router had issues). Is it going to be fixed in 1.8.10 (or 1.8.9.1) ? Thanks, Alex.</p> <p>We occasionally run into this issue. I see the patch set has been rebased at <a href="http://review.whamcloud.com/#change,5971" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5971</a> - is it only waiting approval at this point?</p> <p>We see this issue also at a customer where we deployed servers with Lustre 2.5.0 and his most stable setup seems to be with 1.8.9 clients. Up to this bug!</p> <p>Looks like we ran into this bug as well. As far as I can tell, we hit this on a (1.8) client after it had some network issues of unknown type... Is the patch ok to use as it is?</p> <p>We are using it in production and haven't noticed problems.</p> <p>No plans for further 1.8.x releases</p> Duplicate LU-4452 Development Rank 1|hzvmnz: Rank (Obsolete) 7466 Severity