https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8510 Lustre <p>The following call stack during autotesting on Maloo for <a href="http://review.whamcloud.com/#/c/20546/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/20546/</a>. My new test is standing up 3 MDTs with non-consecutive indices and a couple of OSTs. The method I am using to start the "custom" filesystem seems to be consistent with how other tests start their "custom" filesystems.</p> <p>Link to the Maloo test session results is <a href="https://testing.hpdd.intel.com/test_sessions/4599d8d8-6108-11e6-906c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/4599d8d8-6108-11e6-906c-5254006e85c2</a>.</p> <p>The LBUG is preventing the filesystem from coming up. Any suggestions?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LustreError: 21374:0:(dt_object.h:2633:dt_invalidate()) ASSERTION( dt-&gt;do_ops-&gt;do_invalidate ) failed: LustreError: 21374:0:(dt_object.h:2633:dt_invalidate()) LBUG Pid: 21374, comm: mdt00_002 Call Trace: [&lt;ffffffffa05e67d3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] [&lt;ffffffffa05e6d75&gt;] lbug_with_loc+0x45/0xc0 [libcfs] [&lt;ffffffffa0ea8fcf&gt;] lod_object_unlock+0x39f/0x440 [lod] [&lt;ffffffffa0f11e1b&gt;] mdd_object_unlock+0x3b/0xd0 [mdd] [&lt;ffffffffa0ddbb62&gt;] mdt_unlock_slaves+0x1a2/0x3c0 [mdt] [&lt;ffffffffa0de3c72&gt;] mdt_md_create+0xb52/0xba0 [mdt] [&lt;ffffffffa0de3e2b&gt;] mdt_reint_create+0x16b/0x350 [mdt] [&lt;ffffffffa0de5330&gt;] mdt_reint_rec+0x80/0x210 [mdt] [&lt;ffffffffa0dc7d62&gt;] mdt_reint_internal+0x5b2/0x9b0 [mdt] [&lt;ffffffffa0dd3077&gt;] mdt_reint+0x67/0x140 [mdt] [&lt;ffffffffa0a69aa5&gt;] tgt_request_handle+0x915/0x1320 [ptlrpc] [&lt;ffffffffa0a15c5b&gt;] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [&lt;ffffffffa0a13818&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [&lt;ffffffffa05f1957&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [&lt;ffffffffa0a19d10&gt;] ptlrpc_main+0xaa0/0x1de0 [ptlrpc] [&lt;ffffffffa0a19270&gt;] ? ptlrpc_main+0x0/0x1de0 [ptlrpc] [&lt;ffffffff810a5aef&gt;] kthread+0xcf/0xe0 [&lt;ffffffff810a5a20&gt;] ? kthread+0x0/0xe0 [&lt;ffffffff816469d8&gt;] ret_from_fork+0x58/0x90 [&lt;ffffffff810a5a20&gt;] ? kthread+0x0/0xe0 Kernel panic - not syncing: LBUG </pre> </div></div> CentOS Linux 7/x86_64 LU-8510

ASSERTION( dt->do_ops->do_invalidate ) failed

Bug Blocker Resolved Fixed Zhenyu Xu Giuseppe Di Natale soak Wed, 17 Aug 2016 18:11:32 +0000 Tue, 19 Mar 2019 15:43:00 +0000 Thu, 8 Sep 2016 04:24:55 +0000 Lustre 2.9.0 Lustre 2.9.0 0 5 <p>Bobijam</p> <p>Could you please assist with this issue</p> <p>Thanks</p> <p>Peter</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/22017" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22017</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8510" title="ASSERTION( dt-&gt;do_ops-&gt;do_invalidate ) failed" class="issue-link" data-issue-key="LU-8510"><del>LU-8510</del></a> dne: set osd_obj_ea_ops::dt_invalidate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a9fae446db68cb1c34f2db949c875f30d5e93980</p> <p>The same error also happened during soak testing of '20160902' (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160902" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160902</a>)<br/> Test cluster configuration:<br/> 4 MDS with 1 MDT / MDS, backend FS formatted with <em>ldiskfs</em> , in active-active HA configuration (node pair affected <tt>lola-<span class="error">&#91;8,9&#93;</span></tt>)<br/> 6 OSS with 4 OST / OSS, backend FS formatted with <em>zfs</em> , n active-active HA configuration </p> <p>Error message is the same beside addresses (see attached file vmcore-dmesg.txt) </p> <p>Sequence of events</p> <ul> <li>2016-09-05 15:48:29 Node <tt>lola-8</tt> crashed before (injected fault) failover of lola-9's resources (mdt11) to lola-8.</li> <li>2016-09-05 15:55:04 lola-8 became available before failover took place</li> <li>2016-09-05 16:25:38,987:fsmgmt.fsmgmt:INFO triggering fault mds_failover (<tt>lola-9</tt>)</li> <li>2016-09-05 16:35:39,398: mdt-1 successful mounted on <tt>lola-8</tt>, but stalled in recovery due to missing MDT/MGT of lola-8</li> <li>The MDT/MDT can't be mounted on the primary node (active-active HA configuration) anymore. The error message reads as: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lola-8 ~]# date ; mount -t lustre -o rw,user_xattr /dev/disk/by-id/dm-name-360080e50002ff4f00000026952013088p1 /mnt/soaked-mdt0 ; date Tue Sep 6 02:04:13 PDT 2016 mount.lustre: mount /dev/mapper/360080e50002ff4f00000026952013088p1 at /mnt/soaked-mdt0 failed: Input/output error Is the MGS running? Tue Sep 6 02:05:14 PDT 2016 [root@lola-8 ~]# lctl debug_kernel /tmp/lustre-log-20160906-020514-mgs-mount-fails </pre> </div></div> <p>(Double checked HW; IB and disk resources are operation and sane)<br/> After manual umount of mdt1 and reboot of node <tt>lola-8</tt>, mdt-0, mdt-1 could be mounted and recovery completed within 2 mins and primary<br/> resource could be switched back to (primary) node <tt>lola-9</tt> again.<br/> This symptom is eventually a different bug that happens only by chance due to the node crash.</p></li> </ul> <p>Attached files: messages, console and vmcore-dmesg.txt of affected node <tt>lola-8</tt>, debug log (mask <tt>-1</tt>) containing debug information of the time interval while executing the mount command specified above.<br/> A crash dump file exists and had been store to <tt>lhn.hpdd.intel.com:/scratch/crashdumps/lu-8510/lola-8/127.0.0.1-2016-09-05-15:48:29</tt>.</p> <p>The soak test was executed with el6.7 build (<a href="https://build.hpdd.intel.com/job/lustre-master/3431/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-master/3431/</a> tag 2.8.57)</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22017/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22017/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8510" title="ASSERTION( dt-&gt;do_ops-&gt;do_invalidate ) failed" class="issue-link" data-issue-key="LU-8510"><del>LU-8510</del></a> dne: set osd_obj_ea_ops::dt_invalidate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 13364590a8c9ef64320f62b9937c01aaa6b6fa85</p> <p>Landed for 2.9</p> Related Development Rank 1|hzyl33: Rank (Obsolete) 9223372036854775807 Severity