https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9045 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/881f72c6-e2ae-11e6-bf0a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/881f72c6-e2ae-11e6-bf0a-5254006e85c2</a>.</p> <p>The sub-test test_32c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Panic seen on MDS1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>0:44:09:[15158.281966] LustreError: 27900:0:(osd_handler.c:1562:osd_trans_commit_cb()) ASSERTION( dcb-&gt;dcb_magic == TRANS_COMMIT_CB_MAGIC ) failed: commit callback entry: magic=0 name='tgt_cb_last_committed' 00:44:09:[15158.285311] LustreError: 27900:0:(osd_handler.c:1562:osd_trans_commit_cb()) LBUG 00:44:09:[15158.286863] Pid: 27900, comm: jbd2/loop1-8 00:44:09:[15158.288497] 00:44:09:[15158.288497] Call Trace: 00:44:09:[15158.290800] [&lt;ffffffffa06727f3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] 00:44:09:[15158.292320] [&lt;ffffffffa0672861&gt;] lbug_with_loc+0x41/0xb0 [libcfs] 00:44:09:[15158.293952] [&lt;ffffffffa0f67588&gt;] osd_trans_commit_cb+0x308/0x380 [osd_ldiskfs] 00:44:09:[15158.295497] [&lt;ffffffffa0efb554&gt;] ldiskfs_journal_commit_callback+0x84/0xc0 [ldiskfs] 00:44:09:[15158.297241] [&lt;ffffffffa018260b&gt;] jbd2_journal_commit_transaction+0x161b/0x19a0 [jbd2] 00:44:09:[15158.298788] [&lt;ffffffff81029569&gt;] ? __switch_to+0xd9/0x4c0 00:44:09:[15158.300346] [&lt;ffffffffa0186e99&gt;] kjournald2+0xc9/0x260 [jbd2] 00:44:09:[15158.301683] [&lt;ffffffff810b1720&gt;] ? autoremove_wake_function+0x0/0x40 00:44:09:[15158.303246] [&lt;ffffffffa0186dd0&gt;] ? kjournald2+0x0/0x260 [jbd2] 00:44:09:[15158.304571] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 00:44:09:[15158.305997] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 00:44:09:[15158.307375] [&lt;ffffffff81696958&gt;] ret_from_fork+0x58/0x90 00:44:09:[15158.308660] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 00:44:09:[15158.310084] 00:44:09:[15158.311178] Kernel panic - not syncing: LBUG 00:44:09:[15158.312166] CPU: 0 PID: 27900 Comm: jbd2/loop1-8 Tainted: G OE ------------ 3.10.0-514.6.1.el7_lustre.x86_64 #1 00:44:09:[15158.312166] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 00:44:09:[15158.312166] ffffffffa068fccc 00000000bedb0a5e ffff880039d43b80 ffffffff816863f8 00:44:09:[15158.312166] ffff880039d43c00 ffffffff8167f823 ffffffff00000008 ffff880039d43c10 00:44:09:[15158.312166] ffff880039d43bb0 00000000bedb0a5e 00000000bedb0a5e ffff88007fc0f838 00:44:09:[15158.312166] Call Trace: 00:44:09:[15158.312166] [&lt;ffffffff816863f8&gt;] dump_stack+0x19/0x1b 00:44:09:[15158.312166] [&lt;ffffffff8167f823&gt;] panic+0xe3/0x1f2 00:44:09:[15158.312166] [&lt;ffffffffa0672879&gt;] lbug_with_loc+0x59/0xb0 [libcfs] 00:44:09:[15158.312166] [&lt;ffffffffa0f67588&gt;] osd_trans_commit_cb+0x308/0x380 [osd_ldiskfs] 00:44:09:[15158.312166] [&lt;ffffffffa0efb554&gt;] ldiskfs_journal_commit_callback+0x84/0xc0 [ldiskfs] 00:44:09:[15158.312166] [&lt;ffffffffa018260b&gt;] jbd2_journal_commit_transaction+0x161b/0x19a0 [jbd2] 00:44:09:[15158.312166] [&lt;ffffffff81029569&gt;] ? __switch_to+0xd9/0x4c0 00:44:09:[15158.312166] [&lt;ffffffffa0186e99&gt;] kjournald2+0xc9/0x260 [jbd2] 00:44:09:[15158.312166] [&lt;ffffffff810b1720&gt;] ? wake_up_atomic_t+0x30/0x30 00:44:09:[15158.312166] [&lt;ffffffffa0186dd0&gt;] ? commit_timeout+0x10/0x10 [jbd2] 00:44:09:[15158.312166] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 00:44:09:[15158.312166] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 00:44:09:[15158.312166] [&lt;ffffffff81696958&gt;] ret_from_fork+0x58/0x90 00:44:09:[15158.312166] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>Info required for matching: conf-sanity 32c</p> LU-9045

conf-sanity test_32c: test failed to respond and timed out

Bug Blocker Resolved Fixed WC Triage Maloo Wed, 25 Jan 2017 14:22:33 +0000 Wed, 8 Feb 2017 22:35:13 +0000 Mon, 30 Jan 2017 16:26:13 +0000 Lustre 2.10.0 Lustre 2.10.0 0 9 <p>The LU tickets that landed recently that could be a cause here are <a href="https://jira.whamcloud.com/browse/LU-8821" title="double find in mdt_path_current()" class="issue-link" data-issue-key="LU-8821"><del>LU-8821</del></a>, <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a>, <a href="https://jira.whamcloud.com/browse/LU-8840" title="sanity-lfsck test_2e: @@@@@@ FAIL: (5) Fail to repair crashed linkEA: 0" class="issue-link" data-issue-key="LU-8840"><del>LU-8840</del></a>, and <a href="https://jira.whamcloud.com/browse/LU-8753" title="Recovery already passed deadline with DNE" class="issue-link" data-issue-key="LU-8753"><del>LU-8753</del></a>.</p> <p>Bob, would you be able to cook a revert patch of these landings from recent landings and test again to see if the various test_32c failures go away before we do a full on git bisect to find the cause?</p> <p>I just submitted 6 patches to revert those commits and would vet the test results. The tip of the patch series is <a href="https://review.whamcloud.com/25111" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25111</a>.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4c758afe-e3c5-11e6-9069-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4c758afe-e3c5-11e6-9069-5254006e85c2</a></p> <p>Test results showed that the following commit is the root cause:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 555d02f47401340182b47b3245a657b52fc3e68a Author: Fan Yong &lt;fan.yong@intel.com&gt; Date: Thu Sep 22 16:54:55 2016 +0800 LU-8840 osp: handle EA cache properly </pre> </div></div> <p>Thank you <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a> for the quick root cause identification.</p> <p>The revert patch of the above commit is at <a href="https://review.whamcloud.com/#/c/25134/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25134/</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25134/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25134/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9045" title="conf-sanity test_32c: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9045"><del>LU-9045</del></a> osp: Revert "<a href="https://jira.whamcloud.com/browse/LU-8840" title="sanity-lfsck test_2e: @@@@@@ FAIL: (5) Fail to repair crashed linkEA: 0" class="issue-link" data-issue-key="LU-8840"><del>LU-8840</del></a> osp: handle EA cache properly"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: db1ef0a322f41314abd37b5ec4ad153d63c9b405</p> <p>The revert of the <a href="https://jira.whamcloud.com/browse/LU-8840" title="sanity-lfsck test_2e: @@@@@@ FAIL: (5) Fail to repair crashed linkEA: 0" class="issue-link" data-issue-key="LU-8840"><del>LU-8840</del></a> patch has resolved these failures on master.</p> Related LU-9048 LU-8840 Development Rank 1|hzz1n3: Rank (Obsolete) 9223372036854775807 Severity