https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1951 Lustre <p>MDS crash dumped, attempting to locate dump at this time.<br/> Message from MDS:</p> <p>2012-09-16 11:35:57 LustreError: 5503:0:(osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:<br/> 2012-09-16 11:35:57 LustreError: 5503:0:(osd_handler.c:2343:osd_object_ref_del()) LBUG</p> <p>This looks like a possible dup of ORI-577, however that bug was supposed to have been fixed. </p> <p>MDS did not dump a stack, was configured with panic_on_lbug.<br/> Will attempt to replicate </p> SWL Hyperion/LLNL LU-1951

SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode->i_nlink > 0 ) failed:

Bug Critical Resolved Duplicate Hongchao Zhang Cliff White Sun, 16 Sep 2012 14:47:09 +0000 Tue, 16 Apr 2013 02:38:17 +0000 Tue, 16 Apr 2013 02:38:17 +0000 Lustre 2.3.0 Lustre 2.4.0 0 12 <p>Okay, have vmcore here is the LBUG stack.</p> <p>Pid: 5503, comm: mdt01_012</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0397905&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0397f17&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa100c6a1&gt;&#93;</span> osd_object_ref_del+0x1d1/0x210 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ef109d&gt;&#93;</span> mdo_ref_del+0xad/0xb0 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ef6715&gt;&#93;</span> mdd_unlink+0x815/0xd40 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a61e4&gt;&#93;</span> ? lustre_msg_get_versions+0xa4/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06a7037&gt;&#93;</span> cml_unlink+0x97/0x200 <span class="error">&#91;cmm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f7b454&gt;&#93;</span> mdt_reint_unlink+0x634/0x9e0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f78151&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f719aa&gt;&#93;</span> mdt_reint_internal+0x50a/0x810 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f71cf4&gt;&#93;</span> mdt_reint+0x44/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f65802&gt;&#93;</span> mdt_handle_common+0x922/0x1740 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f666f5&gt;&#93;</span> mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08b499d&gt;&#93;</span> ptlrpc_server_handle_request+0x40d/0xea0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08abf37&gt;&#93;</span> ? ptlrpc_wait_event+0xa7/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03a85b1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa08b5f89&gt;&#93;</span> ptlrpc_main+0xb59/0x1860 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08b5430&gt;&#93;</span> ? ptlrpc_main+0x0/0x1860 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa08b5430&gt;&#93;</span> ? ptlrpc_main+0x0/0x1860 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08b5430&gt;&#93;</span> ? ptlrpc_main+0x0/0x1860 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Kernel panic - not syncing: LBUG</p> <p>Hongchao</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>this issue is different from ORI-577, in which is the local object(say, llog file) that triggers the assertion, and this one<br/> is due to normal file/object during unlink it.</p> <p>this issue could be caused by LDLM problem, for MDT uses LDLM locks to protect parallel directory lock, </p> <p>Hi Cliff,<br/> could you please test it with the debug patch <a href="http://review.whamcloud.com/#change,4009" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4009</a>, <br/> which disables "MDD_DISABLE_PDO_LOCK" in mdd/mdd_internal.h, so we can test whether it is related to LDLM or not, thanks!</p> <p>The patch does not build. I have a vmcore from the crash - would that be useful?</p> <p>okay, thanks!<br/> I'm also trying to create an updated debug patch to collect some logs to help track this issue.</p> <p>The core dump is on brent.whamcloud.com ~/cliffw/lu1951/vmcore-lu1951.gz</p> <p>the possible patch is tracked at <a href="http://review.whamcloud.com/#change,4041" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4041</a></p> <p>Hi Cliff, Could you please test it with this patch? thanks!</p> <p>We have found memory corruption in <a href="https://jira.whamcloud.com/browse/LU-1976" title="SWL - mds hard crash " class="issue-link" data-issue-key="LU-1976"><del>LU-1976</del></a>, the crashed OI index node may cause accessing unknown RAM areas. Under such case, some objects/inodes may be affected also. Although there is no directly evidence to say that <a href="https://jira.whamcloud.com/browse/LU-1951" title="SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:" class="issue-link" data-issue-key="LU-1951"><del>LU-1951</del></a> is just the duplication of <a href="https://jira.whamcloud.com/browse/LU-1976" title="SWL - mds hard crash " class="issue-link" data-issue-key="LU-1976"><del>LU-1976</del></a>, I tend to the duplication of <a href="https://jira.whamcloud.com/browse/LU-1976" title="SWL - mds hard crash " class="issue-link" data-issue-key="LU-1976"><del>LU-1976</del></a>. Since the patch for <a href="https://jira.whamcloud.com/browse/LU-1976" title="SWL - mds hard crash " class="issue-link" data-issue-key="LU-1976"><del>LU-1976</del></a> is ready, we can wait for the new result with such patch to check whether <a href="https://jira.whamcloud.com/browse/LU-1951" title="SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:" class="issue-link" data-issue-key="LU-1951"><del>LU-1951</del></a> can be reproduced or not.</p> <p>Dropping priority as this only occurred once in five days of testing</p> <p>Looks like we still have this issue with build 24:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ep 27 21:56:35 hyperion-rst6 kernel: LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed: Sep 27 21:56:35 hyperion-rst6 kernel: LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed: Sep 27 21:56:35 hyperion-rst6 kernel: LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) LBUG Sep 27 21:56:35 hyperion-rst6 kernel: LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) LBUG Sep 27 21:56:35 hyperion-rst6 kernel: Pid: 5611, comm: mdt00_015 Sep 27 21:56:35 hyperion-rst6 kernel: Sep 27 21:56:35 hyperion-rst6 kernel: Call Trace: Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0392905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0392f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0a946a1&gt;] osd_object_ref_del+0x1d1/0x210 [osd_ldiskfs] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0efa09d&gt;] mdo_ref_del+0xad/0xb0 [mdd] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0eff715&gt;] mdd_unlink+0x815/0xdb0 [mdd] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa09581e4&gt;] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa08bd037&gt;] cml_unlink+0x97/0x200 [cmm] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f83ddf&gt;] ? mdt_version_get_save+0x8f/0xd0 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f84454&gt;] mdt_reint_unlink+0x634/0x9e0 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f81151&gt;] mdt_reint_rec+0x41/0xe0 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f7a9aa&gt;] mdt_reint_internal+0x50a/0x810 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f7acf4&gt;] mdt_reint+0x44/0xe0 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f6e802&gt;] mdt_handle_common+0x922/0x1740 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0f6f6f5&gt;] mdt_regular_handle+0x15/0x20 [mdt] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0966b3c&gt;] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa039365e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa03a513f&gt;] ? lc_watchdog_touch+0x6f/0x180 [libcfs] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa095df37&gt;] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffff810533f3&gt;] ? __wake_up+0x53/0x70 Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0968111&gt;] ptlrpc_main+0xbf1/0x19e0 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 27 21:56:35 hyperion-rst6 kernel: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Sep 27 21:56:35 hyperion-rst6 kernel: Sep 27 21:56:35 hyperion-rst6 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Also, some issues when recovering the MDS after the dump:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Lustre: lustre-MDT0000: used disk, loading Lustre: 4204:0:(ldlm_lib.c:2139:target_recovery_init()) RECOVERY: service lustre-MDT0000, 91 recoverable clients, last_transno 8624944131 Lustre: 4140:0:(mgc_request.c:1534:mgc_process_recover_log()) <span class="code-object">Process</span> recover log lustre-mdtir error -22 Lustre: lustre-MDT0000: Will be in recovery <span class="code-keyword">for</span> at least 5:00, or until 91 clients reconnect LustreError: 4216:0:(mds_lov.c:351:mds_lov_update_objids()) Unexpected gap in objids LustreError: 4216:0:(mdt_recovery.c:497:mdt_txn_stop_cb()) Replay transno 8624998654 failed: rc -39 LustreError: 4216:0:(mds_lov.c:351:mds_lov_update_objids()) Unexpected gap in objids Lustre: lustre-MDT0000: disconnecting 1 stale clients Lustre: lustre-MDT0000: Recovery over after 1:22, of 91 clients 90 recovered and 1 was evicted. Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST001b_UUID now active, resetting orphans Lustre: Skipped 5 previous similar messages Lustre: Skipped 45 previous similar messages </pre> </div></div> <p>/var/log/messages from evicted client</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 27 23:13:24 hyperion787 kernel: Lustre: 3928:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1348812749/real 1348812749] req@ffff8802116cec00 x1414295663929201/t0(0) o250-&gt;MGC192.168.127.6@o2ib@192.168.127.6@o2ib:26/25 lens 400/544 e 0 to 1 dl 1348812804 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 27 23:13:24 hyperion787 kernel: Lustre: 3928:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Sep 27 23:14:59 hyperion787 kernel: Lustre: Evicted from MGS (at MGC192.168.127.6@o2ib_0) after server handle changed from 0x5e1536afc64a7a2d to 0x9107144105607180 Sep 27 23:14:59 hyperion787 kernel: Lustre: MGC192.168.127.6@o2ib: Reactivating <span class="code-keyword">import</span> Sep 27 23:14:59 hyperion787 kernel: Lustre: MGC192.168.127.6@o2ib: Connection restored to MGS (at 192.168.127.6@o2ib) Sep 27 23:16:21 hyperion787 kernel: Lustre: lustre-MDT0000-mdc-ffff88033ba61400: Connection restored to lustre-MDT0000 (at 192.168.127.6@o2ib1) </pre> </div></div> <p>Is the patch (<a href="http://review.whamcloud.com/#change,4041" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4041</a>) included in this build(24)?</p> <p>No it is not.</p> <p>Hongchao</p> <p>If you think that it should be then please set inspectors on the patch.</p> <p>Thanks</p> <p>Peter</p> <p>MDS crashed, panic stack:<br/> 2012-09-27 21:56:35 LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:<br/> 2012-09-27 21:56:35 LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) LBUG<br/> 2012-09-27 21:56:35 Pid: 5611, comm: mdt00_015<br/> 2012-09-27 21:56:35<br/> 2012-09-27 21:56:35 Sep 27 21:56:35 Call Trace:<br/> 2012-09-27 21:56:35 hyperion-rst6 ke <span class="error">&#91;&lt;ffffffffa0392905&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:35 rnel: LustreErro <span class="error">&#91;&lt;ffffffffa0392f17&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:35 r: 5611:0:(osd_h <span class="error">&#91;&lt;ffffffffa0a946a1&gt;&#93;</span> osd_object_ref_del+0x1d1/0x210 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2012-09-27 21:56:35 andler.c:2343:os <span class="error">&#91;&lt;ffffffffa0efa09d&gt;&#93;</span> mdo_ref_del+0xad/0xb0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-27 21:56:35 d_object_ref_del <span class="error">&#91;&lt;ffffffffa0eff715&gt;&#93;</span> mdd_unlink+0x815/0xdb0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-27 21:56:35 ()) ASSERTION( i <span class="error">&#91;&lt;ffffffffa09581e4&gt;&#93;</span> ? lustre_msg_get_versions+0xa4/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 node-&gt;i_nlink &gt; <span class="error">&#91;&lt;ffffffffa08bd037&gt;&#93;</span> cml_unlink+0x97/0x200 <span class="error">&#91;cmm&#93;</span><br/> 2012-09-27 21:56:35 0 ) failed:<br/> 2012-09-27 21:56:35 Sep <span class="error">&#91;&lt;ffffffffa0f83ddf&gt;&#93;</span> ? mdt_version_get_save+0x8f/0xd0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 27 21:56:35 hyp <span class="error">&#91;&lt;ffffffffa0f84454&gt;&#93;</span> mdt_reint_unlink+0x634/0x9e0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 erion-rst6 kerne <span class="error">&#91;&lt;ffffffffa0f81151&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 l: LustreError: <span class="error">&#91;&lt;ffffffffa0f7a9aa&gt;&#93;</span> mdt_reint_internal+0x50a/0x810 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 5611:0:(osd_hand <span class="error">&#91;&lt;ffffffffa0f7acf4&gt;&#93;</span> mdt_reint+0x44/0xe0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 ler.c:2343:osd_o <span class="error">&#91;&lt;ffffffffa0f6e802&gt;&#93;</span> mdt_handle_common+0x922/0x1740 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 bject_ref_del()) <span class="error">&#91;&lt;ffffffffa0f6f6f5&gt;&#93;</span> mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 LBUG<br/> 2012-09-27 21:56:35 LBUG<br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0966b3c&gt;&#93;</span> ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa039365e&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa03a513f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x180 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa095df37&gt;&#93;</span> ? ptlrpc_wait_event+0xa7/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0968111&gt;&#93;</span> ptlrpc_main+0xbf1/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 2012-09-27 21:56:35<br/> 2012-09-27 21:56:35 Kernel panic - not syncing: LBUG<br/> 2012-09-27 21:56:35 Pid: 5611, comm: mdt00_015 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.x86_64 #1<br/> 2012-09-27 21:56:35 Sep 27 21:56:35 Call Trace:<br/> 2012-09-27 21:56:35 hyperion-rst6 ke <span class="error">&#91;&lt;ffffffff814fd58a&gt;&#93;</span> ? panic+0xa0/0x168<br/> 2012-09-27 21:56:35 rnel: Kernel pan <span class="error">&#91;&lt;ffffffffa0392f6b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:35 ic - not syncing <span class="error">&#91;&lt;ffffffffa0a946a1&gt;&#93;</span> ? osd_object_ref_del+0x1d1/0x210 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2012-09-27 21:56:35 : LBUG<br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0efa09d&gt;&#93;</span> ? mdo_ref_del+0xad/0xb0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0eff715&gt;&#93;</span> ? mdd_unlink+0x815/0xdb0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa09581e4&gt;&#93;</span> ? lustre_msg_get_versions+0xa4/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa08bd037&gt;&#93;</span> ? cml_unlink+0x97/0x200 <span class="error">&#91;cmm&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f83ddf&gt;&#93;</span> ? mdt_version_get_save+0x8f/0xd0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f84454&gt;&#93;</span> ? mdt_reint_unlink+0x634/0x9e0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f81151&gt;&#93;</span> ? mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f7a9aa&gt;&#93;</span> ? mdt_reint_internal+0x50a/0x810 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f7acf4&gt;&#93;</span> ? mdt_reint+0x44/0xe0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:35 <span class="error">&#91;&lt;ffffffffa0f6e802&gt;&#93;</span> ? mdt_handle_common+0x922/0x1740 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0f6f6f5&gt;&#93;</span> ? mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0966b3c&gt;&#93;</span> ? ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa039365e&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa03a513f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x180 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa095df37&gt;&#93;</span> ? ptlrpc_wait_event+0xa7/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0968111&gt;&#93;</span> ? ptlrpc_main+0xbf1/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffffa0967520&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-27 21:56:36 <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>vmcore is on brent in ~/cliffw/lu1948/erofs</p> <p>Check the last comment posted by Cliff on <a href="https://jira.whamcloud.com/browse/LU-1948" title="ldiskfs - MDS goes read-only (SWL)" class="issue-link" data-issue-key="LU-1948"><del>LU-1948</del></a>, we hit this again while running SWL. </p> <p>Hit this again while running SWL, backtrace:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 4891 TASK: ffff88016dedaaa0 CPU: 13 COMMAND: <span class="code-quote">"mdt03_014"</span> #0 [ffff88016e693918] machine_kexec at ffffffff8103281b #1 [ffff88016e693978] crash_kexec at ffffffff810ba792 #2 [ffff88016e693a48] panic at ffffffff814fd591 #3 [ffff88016e693ac8] lbug_with_loc at ffffffffa0395f6b [libcfs] #4 [ffff88016e693ae8] osd_object_ref_del at ffffffffa0a8b6c1 [osd_ldiskfs] #5 [ffff88016e693b18] mdo_ref_del at ffffffffa0ef0ffd [mdd] #6 [ffff88016e693b28] mdd_unlink at ffffffffa0ef6675 [mdd] #7 [ffff88016e693be8] cml_unlink at ffffffffa06bc037 [cmm] #8 [ffff88016e693c28] mdt_reint_unlink at ffffffffa0f7b454 [mdt] #9 [ffff88016e693ca8] mdt_reint_rec at ffffffffa0f78151 [mdt] #10 [ffff88016e693cc8] mdt_reint_internal at ffffffffa0f719aa [mdt] #11 [ffff88016e693d18] mdt_reint at ffffffffa0f71cf4 [mdt] #12 [ffff88016e693d38] mdt_handle_common at ffffffffa0f65802 [mdt] #13 [ffff88016e693d88] mdt_regular_handle at ffffffffa0f666f5 [mdt] #14 [ffff88016e693d98] ptlrpc_server_handle_request at ffffffffa095db3c [ptlrpc] #15 [ffff88016e693e98] ptlrpc_main at ffffffffa095f111 [ptlrpc] #16 [ffff88016e693f48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>The lustre debug dump log. Though not much useful for this LBUG. But it seems there are some lnet error, Liang, could you please have a look? </p> <p>I found something suspicious in mdd_rename(), but I'm not expert of this, so please check this for me:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* Remove old target object * For tobj is remote case cmm layer has processed * and set tobj to NULL then. So when tobj is NOT NULL, * it must be local one. */ if (tobj &amp;&amp; mdd_object_exists(mdd_tobj)) { mdd_write_lock(env, mdd_tobj, MOR_TGT_CHILD); if (mdd_is_dead_obj(mdd_tobj)) { mdd_write_unlock(env, mdd_tobj); /* shld not be dead, something is wrong */ CERROR("tobj is dead, something is wrong\n"); rc = -EINVAL; goto cleanup; } mdo_ref_del(env, mdd_tobj, handle); /* Remove dot reference. */ if (is_dir) mdo_ref_del(env, mdd_tobj, handle); la-&gt;la_valid = LA_CTIME; rc = mdd_attr_check_set_internal(env, mdd_tobj, la, handle, 0); if (rc) GOTO(fixup_tpobj, rc); rc = mdd_finish_unlink(env, mdd_tobj, ma, handle); mdd_write_unlock(env, mdd_tobj); if (rc) GOTO(fixup_tpobj, rc); </pre> </div></div> <p>If mdd_attr_check_set_internal() or mdd_finish_unlink() failed, it will try to revert changes by re-inserting @mdd_tobj into @mdd_tpobj again without fix refcount of @mdd_tobj:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fixup_tpobj: if (rc) { rc2 = __mdd_index_delete(env, mdd_tpobj, tname, is_dir, handle, BYPASS_CAPA); if (rc2) CWARN("tp obj fix error %d\n",rc2); if (mdd_tobj &amp;&amp; mdd_object_exists(mdd_tobj) &amp;&amp; !mdd_is_dead_obj(mdd_tobj)) { rc2 = __mdd_index_insert(env, mdd_tpobj, mdo2fid(mdd_tobj), tname, is_dir, handle, BYPASS_CAPA); if (rc2) CWARN("tp obj fix error %d\n",rc2); } } </pre> </div></div> <p>So if everything got reverted, refcount on target object will be wrong.<br/> Is this analysis correct? </p> <p>I've posted another patch for this: <a href="http://review.whamcloud.com/#change,4197" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4197</a><br/> it should have fixed something but not sure if it can fix this bug.</p> <p>Liang, probably makes sense to set add CERROR() to see whether we hit this path.</p> <p>Dropping priority because landed for 2.3</p> <p>Liang, Oleg, what about <a href="http://review.whamcloud.com/4136?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4136?</a> That patch didn't land to b2_3. Was that intended to fix the MDS crash, or is it a secondary problem that doesn't need to be fixed for 2.3.0?</p> <p>I think the problem fixed by review-4136 existed for long time (since 2.1), so probably it's not the reason of the crash here, but we should land it to master at least. </p> <p>patch for master: <a href="http://review.whamcloud.com/4405" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4405</a><br/> port of <a href="http://review.whamcloud.com/#change,4197" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4197</a>. Context too different to just be cherry picked directly.</p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-3022" title="osd_handler.c:2534:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed" class="issue-link" data-issue-key="LU-3022"><del>LU-3022</del></a></p> Development Rank 1|hzv573: Rank (Obsolete) 4375 Severity