https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12571 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/d79edebe-abc3-11e9-a0be-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d79edebe-abc3-11e9-a0be-52540065bddc</a></p> <p>test_150 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-70vm1 crashed during sanity-dom test_150 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 19240:0:(lu_object.c:1196:lu_device_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1 LustreError: 19240:0:(lu_object.c:1196:lu_device_fini()) LBUG Pid: 19240, comm: umount 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 Call Trace: [&lt;ffffffffc06658bc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [&lt;ffffffffc066596c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [&lt;ffffffffc07d293b&gt;] lu_device_fini+0xbb/0xc0 [obdclass] [&lt;ffffffffc0c98994&gt;] lovsub_device_free+0x94/0x170 [lov] [&lt;ffffffffc07d7828&gt;] lu_stack_fini+0x98/0xe0 [obdclass] [&lt;ffffffffc07db89e&gt;] cl_stack_fini+0xe/0x10 [obdclass] [&lt;ffffffffc0c750dc&gt;] lov_device_fini+0x7c/0x170 [lov] [&lt;ffffffffc07d77eb&gt;] lu_stack_fini+0x5b/0xe0 [obdclass] [&lt;ffffffffc07db89e&gt;] cl_stack_fini+0xe/0x10 [obdclass] [&lt;ffffffffc0d240b0&gt;] cl_sb_fini+0x70/0x1b0 [lustre] [&lt;ffffffffc0cf2688&gt;] ll_put_super+0x528/0x8a0 [lustre] [&lt;ffffffff8a84433d&gt;] generic_shutdown_super+0x6d/0x100 [&lt;ffffffff8a844732&gt;] kill_anon_super+0x12/0x20 [&lt;ffffffffc07c88d5&gt;] lustre_kill_super+0x45/0x50 [obdclass] [&lt;ffffffff8a844aee&gt;] deactivate_locked_super+0x4e/0x70 [&lt;ffffffff8a845276&gt;] deactivate_super+0x46/0x60 [&lt;ffffffff8a86387f&gt;] cleanup_mnt+0x3f/0x80 [&lt;ffffffff8a863912&gt;] __cleanup_mnt+0x12/0x20 [&lt;ffffffff8a6be90b&gt;] task_work_run+0xbb/0xe0 [&lt;ffffffff8a62bc65&gt;] do_notify_resume+0xa5/0xc0 [&lt;ffffffff8ad76134&gt;] int_signal+0x12/0x17 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff Kernel panic - not syncing: LBUG CPU: 0 PID: 19240 Comm: umount Kdump: loaded Tainted: G OE ------------ 3.10.0-957.21.3.el7.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: [&lt;ffffffff8ad63107&gt;] dump_stack+0x19/0x1b [&lt;ffffffff8ad5c810&gt;] panic+0xe8/0x21f [&lt;ffffffffc06659bb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [&lt;ffffffffc07d293b&gt;] lu_device_fini+0xbb/0xc0 [obdclass] [&lt;ffffffffc0c98994&gt;] lovsub_device_free+0x94/0x170 [lov] [&lt;ffffffffc07d7828&gt;] lu_stack_fini+0x98/0xe0 [obdclass] [&lt;ffffffffc07db89e&gt;] cl_stack_fini+0xe/0x10 [obdclass] [&lt;ffffffffc0c750dc&gt;] lov_device_fini+0x7c/0x170 [lov] [&lt;ffffffffc07d77eb&gt;] lu_stack_fini+0x5b/0xe0 [obdclass] [&lt;ffffffffc07db89e&gt;] cl_stack_fini+0xe/0x10 [obdclass] [&lt;ffffffffc0d240b0&gt;] cl_sb_fini+0x70/0x1b0 [lustre] [&lt;ffffffffc0cf2688&gt;] ll_put_super+0x528/0x8a0 [lustre] [&lt;ffffffff8a88ab83&gt;] ? fsnotify_destroy_marks+0x13/0xc0 [&lt;ffffffff8a753f8d&gt;] ? call_rcu_sched+0x1d/0x20 [&lt;ffffffffc0d1945c&gt;] ? ll_destroy_inode+0x1c/0x20 [lustre] [&lt;ffffffff8a85efdb&gt;] ? destroy_inode+0x3b/0x60 [&lt;ffffffff8a85f115&gt;] ? evict+0x115/0x180 [&lt;ffffffff8a85f1be&gt;] ? dispose_list+0x3e/0x50 [&lt;ffffffff8a85fe7b&gt;] ? evict_inodes+0xdb/0x140 [&lt;ffffffff8a84433d&gt;] generic_shutdown_super+0x6d/0x100 [&lt;ffffffff8a844732&gt;] kill_anon_super+0x12/0x20 [&lt;ffffffffc07c88d5&gt;] lustre_kill_super+0x45/0x50 [obdclass] [&lt;ffffffff8a844aee&gt;] deactivate_locked_super+0x4e/0x70 [&lt;ffffffff8a845276&gt;] deactivate_super+0x46/0x60 [&lt;ffffffff8a86387f&gt;] cleanup_mnt+0x3f/0x80 [&lt;ffffffff8a863912&gt;] __cleanup_mnt+0x12/0x20 [&lt;ffffffff8a6be90b&gt;] task_work_run+0xbb/0xe0 [&lt;ffffffff8a62bc65&gt;] do_notify_resume+0xa5/0xc0 [&lt;ffffffff8ad76134&gt;] int_signal+0x12/0x17 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-dom test_150 - trevis-70vm1 crashed during sanity-dom test_150</p> LU-12571

sanity-dom test_150: ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1

Bug Minor Open Unresolved WC Triage Maloo Mon, 22 Jul 2019 04:29:08 +0000 Tue, 28 Jan 2020 07:06:55 +0000 Lustre 2.13.0 0 9 <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/53c01e8a-eb12-11e9-a197-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/53c01e8a-eb12-11e9-a197-52540065bddc</a><br/> six other times in the past 4 weeks, so definitely not very common.</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/30ed80fc-0d55-4032-8ed6-b287376bedce" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/30ed80fc-0d55-4032-8ed6-b287376bedce</a></p> <p>+1 on master @<a href="https://testing.whamcloud.com/test_sessions/431dfbf0-778f-46c2-84d5-e61058e669d6" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/431dfbf0-778f-46c2-84d5-e61058e669d6</a></p> <p>Maybe +1 on master @<a href="https://testing.whamcloud.com/test_sets/8b37d52e-ecfc-11e9-a197-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8b37d52e-ecfc-11e9-a197-52540065bddc</a> in that maloo reported that test sanity-dom test_150 crashed. I don't see any links to console logs etc. so I do not see how I can verify what actually happened on the nodes to confirm a match.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a> the console logs for the <a href="https://testing.whamcloud.com/test_sets/8b37d52e-ecfc-11e9-a197-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">test session</a> are linked on the right side of the orange test "info box" at the top, not on a per-subtest basis anymore. <br/> <span class="image-wrap" style=""><a id="34006_thumb" href="https://jira.whamcloud.com/secure/attachment/34006/34006_Screen+Shot+2019-12-10+at+20.19.21.png" title="Screen Shot 2019-12-10 at 20.19.21.png" file-preview-type="image" file-preview-id="34006" file-preview-title="Screen Shot 2019-12-10 at 20.19.21.png"><img src="https://jira.whamcloud.com/secure/thumbnail/34006/_thumb_34006.png" style="border: 0px solid black" role="presentation"/></a></span></p> <p>Selecting "<tt>Show</tt>" for "<tt>console.trevis-70vm1.log</tt>" to view the console log, and searching for "<tt>sanity test 150</tt>" brings me to the failed subtest, which indeed shows that this is the same issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 6344:0:(lu_object.c:1265:lu_device_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1 LustreError: 6344:0:(lu_object.c:1265:lu_device_fini()) LBUG </pre> </div></div> <p>another hit <a href="https://testing.whamcloud.com/test_sets/99598020-3974-11ea-b1e8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/99598020-3974-11ea-b1e8-52540065bddc</a></p> <p>Though crashed in test 150, that is not this test issue, the <tt>remount_client</tt> just reveals object with non-released reference. Log analysis shows that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [19380.752462] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 11:00:54 (1563706854) &lt;&lt;&lt;&lt; skipped &gt;&gt;&gt;&gt; [19384.237133] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 11:00:57 (1563706857) [19384.517120] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) header@ffff9ff5a57472a0[0x0, 3, [0x100000000:0x90ac:0x0] hash]{ [19384.519479] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) ....lovsub@ffff9ff5a5747300[65536] [19384.521385] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) ....osc@ffff9ff5b9c95a00id: 0x0:37036 idx: 0 gen: 0 kms_valid: 1 kms 2097152 rc: 0 force_sync: 0 min_xid: 0 size: 2097152 mtime: 1563706854 atime: 0 ctime: 1563706854 blocks: 1 [19384.525535] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) } header@ffff9ff5a57472a0 [19384.527762] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) header@ffff9ff5a57471f8[0x0, 1, [0x2000470d1:0xe6:0x0] hash]{ [19384.530080] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) ....lovsub@ffff9ff5a5747258[0] [19384.531918] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) ....mdc@ffff9ff5b9c95b40id: 0x2000470d1:230 idx: 0 gen: 0 kms_valid: 1 kms 0 rc: 0 force_sync: 0 min_xid: 0 size: 0 mtime: 1563706854 atime: 1563706854 ctime: 1563706854 blocks: 2 [19384.536024] LustreError: 19240:0:(lovsub_dev.c:94:lovsub_device_free()) } header@ffff9ff5a57471f8 [19384.540415] LustreError: 19240:0:(lu_object.c:1196:lu_device_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1 </pre> </div></div> <p>That object with extra reference is from test 131e, <tt>ctime/mtime</tt> indicates that and also object structure: 0-bytes in DOM part and 2MB OSC component, like test 131e does. Nevertheless, I have no idea where it lost reference and why, this needs more investigation</p> <p><a href="https://testing.whamcloud.com/test_sessions/33af08a7-bf95-423a-b01e-9eb7e9bba066" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/33af08a7-bf95-423a-b01e-9eb7e9bba066</a></p> Development Rank 1|i00jzj: Rank (Obsolete) 9223372036854775807 Severity