https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4386 Lustre <p>While running conf-sanity test 50h, the following LBUG failure occurred on MDS node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:07:20:Lustre: Permanently deactivating lustre-OST0000 09:07:20:Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request 09:07:20:LustreError: 8773:0:(import.c:285:ptlrpc_invalidate_import()) ASSERTION( imp-&gt;imp_invalid ) failed: 09:07:20:LustreError: 8773:0:(import.c:285:ptlrpc_invalidate_import()) LBUG 09:07:20:Pid: 8773, comm: llog_process_th 09:07:20: 09:07:20:Call Trace: 09:07:20: [&lt;ffffffffa04ea905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 09:07:20: [&lt;ffffffffa04eaf17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 09:07:21: [&lt;ffffffffa0808421&gt;] ptlrpc_invalidate_import+0x391/0x8d0 [ptlrpc] 09:07:21: [&lt;ffffffffa04fb5f1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 09:07:21: [&lt;ffffffffa0647f59&gt;] ? lprocfs_write_frac_helper+0xa9/0x150 [obdclass] 09:07:21: [&lt;ffffffffa07ddca5&gt;] ptlrpc_set_import_active+0x405/0x6e0 [ptlrpc] 09:07:21: [&lt;ffffffffa09992ae&gt;] osc_wr_active+0xfe/0x120 [osc] 09:07:21: [&lt;ffffffffa065e05e&gt;] class_process_proc_param+0x35e/0x780 [obdclass] 09:07:21: [&lt;ffffffffa098688a&gt;] osc_process_config_base+0x4a/0x60 [osc] 09:07:21: [&lt;ffffffffa099df1f&gt;] osc_cl_process_config+0xaf/0xe0 [osc] 09:07:21: [&lt;ffffffffa065fb53&gt;] class_process_config+0x16d3/0x1c30 [obdclass] 09:07:21: [&lt;ffffffffa0658ef1&gt;] ? lustre_cfg_new+0x391/0x7e0 [obdclass] 09:07:21: [&lt;ffffffffa066115b&gt;] class_config_llog_handler+0x9bb/0x1610 [obdclass] 09:07:22: [&lt;ffffffffa04fb5f1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 09:07:22: [&lt;ffffffffa062a1f8&gt;] llog_process_thread+0x888/0xd00 [obdclass] 09:07:22: [&lt;ffffffffa0629970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 09:07:22: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 09:07:22: [&lt;ffffffffa0629970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 09:07:22: [&lt;ffffffffa0629970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 09:07:22: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 09:07:23: 09:07:23:Kernel panic - not syncing: LBUG </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/838153a6-628c-11e3-bf4a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/838153a6-628c-11e3-bf4a-52540035b04c</a></p> <br/> Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_4/67/">http://build.whamcloud.com/job/lustre-b2_4/67/</a><br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)<br/> LU-4386

conf-sanity test_50h: LBUG: (import.c:285:ptlrpc_invalidate_import()) ASSERTION( imp->imp_invalid ) failed

Bug Blocker Resolved Fixed Hongchao Zhang Jian Yu mn4 Sat, 14 Dec 2013 15:10:10 +0000 Thu, 20 Feb 2014 18:15:29 +0000 Tue, 18 Feb 2014 13:45:20 +0000 Lustre 2.4.1 Lustre 2.5.1 Lustre 2.6.0 Lustre 2.5.1 0 8 <p>This is a regression failure caused by Lustre b2_4 build #67 because the failure did not occur before while running interop testing on previous Lustre b2_4 builds with 2.3.0 server.</p> <p>Hongchao</p> <p>What is your assessment of this issue?</p> <p>Thanks</p> <p>Peter</p> <p>There are two patches included in Lustre b2_4 build #67:<br/> <a href="https://jira.whamcloud.com/browse/LU-4362" title="cfs_hash_rehash_key() passed wrong parameters to cfs_hash_keycpy" class="issue-link" data-issue-key="LU-4362"><del>LU-4362</del></a>: <a href="http://review.whamcloud.com/8510" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8510</a><br/> <a href="https://jira.whamcloud.com/browse/LU-3929" title="2.1.6-&gt;2.4.1 rolling upgrade: lustre-MDT0000: recovery is timed out, evict stale exports" class="issue-link" data-issue-key="LU-3929"><del>LU-3929</del></a>: <a href="http://review.whamcloud.com/8513" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8513</a></p> <p>FYI, I tried to run the interop testing with Lustre 2.3.0 server and the <a href="https://jira.whamcloud.com/browse/LU-3929" title="2.1.6-&gt;2.4.1 rolling upgrade: lustre-MDT0000: recovery is timed out, evict stale exports" class="issue-link" data-issue-key="LU-3929"><del>LU-3929</del></a> build as client, conf-sanity passed twice:<br/> <a href="https://maloo.whamcloud.com/test_sets/5944e50c-6571-11e3-b9b7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5944e50c-6571-11e3-b9b7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/025877d6-6553-11e3-9f05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/025877d6-6553-11e3-9f05-52540035b04c</a></p> <p>More testing is ongoing.</p> <p>In the meantime, please look into the failure to find out the real cause. Thanks!</p> <p>I triggered Lustre b2_4 build #67 client + 2.3.0 server full group test session again, and the test results showed that the failure did not occur in this session:<br/> <a href="https://maloo.whamcloud.com/test_sessions/9ed3c6e4-65c4-11e3-b9b7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/9ed3c6e4-65c4-11e3-b9b7-52540035b04c</a></p> <p>there could be a race between ptlrpc_invalidate_import and ptlrpc_activate_import,</p> <p>1, in llog_process_thread<br/> osc_process_config_base -&gt; class_process_proc_param -&gt; osc_wr_active -&gt; ptlrpc_set_import_active -&gt; ptlrpc_invalidate_import, which will call <br/> ptlrpc_deactivate_import to deactive the import.</p> <p>2, during lod_process_config processed "LCFG_LOV_ADD_INA" (initial inactive), lod_add_device was called to connect the target even "active" parameter was false,<br/> and if the connection is successful, ptlrpc_activate_import will be called in ptlrpc_connect_interpret, and if it is just called between ptlrpc_invalidate_import<br/> and the assertion "LASSERT(imp-&gt;imp_invalid)", then it will trigger the LBUG.</p> <p>FYI, the failure did not occur in the Lustre 2.4.2 RC1 client + 2.3.0 server full group test session:<br/> <a href="https://maloo.whamcloud.com/test_sessions/2f6f4020-67eb-11e3-846d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/2f6f4020-67eb-11e3-846d-52540035b04c</a></p> <p>the issue has been reproduced locally by adding some delay between operations and restarting MDT.<br/> the corresponding patch is under creation and test.</p> <p>the patch against b2_4 is tracked at <a href="http://review.whamcloud.com/#/c/8634/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8634/</a>, and it is also needed in b2_3.</p> <p>Thanks Hongchao. Is this change needed on master too?</p> <p>Yes, the master also need this patch.</p> <p>also hit this issue in master branch under DNE mode</p> <p><a href="https://maloo.whamcloud.com/test_sets/f722c56e-6ec7-11e3-9dbc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f722c56e-6ec7-11e3-9dbc-52540035b04c</a></p> <p>another instance seen in review-dne</p> <p><a href="https://maloo.whamcloud.com/test_sets/7d013360-711a-11e3-b502-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d013360-711a-11e3-b502-52540035b04c</a></p> <p>another instance in master review<br/> <a href="https://maloo.whamcloud.com/test_sets/fb94c47e-75e5-11e3-b3c0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fb94c47e-75e5-11e3-b3c0-52540035b04c</a></p> <p>I cherry-picked this patch to master (using the handy new "Cherry Pick" option in Gerrit), and updated the commit comment slightly.</p> <p><a href="http://review.whamcloud.com/8747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8747</a></p> <p>While verifying patch <a href="http://review.whamcloud.com/8734" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8734</a> on Lustre b2_5 branch, conf-sanity test 50h hit the same failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ===== 09:05:36 (1389027936)^M Lustre: DEBUG MARKER: mkdir -p /mnt/mds1^M Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1^M Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl /dev/lvm-Role_MDS/P1 /mnt/mds1^M LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: ^M Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs.^M Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust^M Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null^M Lustre: MGS: Regenerating lustre-OST0000 log by user request.^M Lustre: Permanently deactivating lustre-OST0000^M Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request^M LustreError: 14612:0:(import.c:284:ptlrpc_invalidate_import()) ASSERTION( imp-&gt;imp_invalid ) failed: ^M LustreError: 14612:0:(import.c:284:ptlrpc_invalidate_import()) LBUG^M Pid: 14612, comm: llog_process_th^M ^M Call Trace:^M [&lt;ffffffffa03ef895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa03efe97&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M [&lt;ffffffffa08ac8c7&gt;] ptlrpc_invalidate_import+0x377/0x880 [ptlrpc]^M [&lt;ffffffffa03ff7b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs]^M [&lt;ffffffffa064f549&gt;] ? lprocfs_write_frac_helper+0xa9/0x150 [obdclass]^M [&lt;ffffffffa08897a8&gt;] ptlrpc_set_import_active+0x3d8/0x670 [ptlrpc]^M [&lt;ffffffffa12b3f0e&gt;] osp_wr_active+0xfe/0x120 [osp]^M [&lt;ffffffffa066eb06&gt;] class_process_proc_param+0x356/0x770 [obdclass]^M [&lt;ffffffffa12a2b45&gt;] osp_process_config+0x2a5/0x640 [osp]^M [&lt;ffffffffa0671560&gt;] class_process_config+0x1910/0x1ad0 [obdclass]^M [&lt;ffffffffa066a352&gt;] ? lustre_cfg_new+0x312/0x6e0 [obdclass]^M [&lt;ffffffffa067283b&gt;] class_config_llog_handler+0xa2b/0x1880 [obdclass]^M [&lt;ffffffffa03ff7b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs]^M [&lt;ffffffffa0634dab&gt;] llog_process_thread+0x8ab/0xd60 [obdclass]^M [&lt;ffffffffa067d43f&gt;] ? keys_fill+0x6f/0x190 [obdclass]^M [&lt;ffffffffa06352a5&gt;] llog_process_thread_daemonize+0x45/0x70 [obdclass]^M [&lt;ffffffffa0635260&gt;] ? llog_process_thread_daemonize+0x0/0x70 [obdclass]^M [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0^M [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20^M [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0^M [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20^M ^M Kernel panic - not syncing: LBUG^M </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/4a0358ae-7736-11e3-9ce8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4a0358ae-7736-11e3-9ce8-52540035b04c</a></p> <p>This is a regression on Lustre b2_5 branch introduced by build #6 or #7 or #8.</p> <p>Hi Yujian, does the build #6, #7, #8 contain the patch at <a href="http://review.whamcloud.com/#/c/8634/?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8634/?</a> Thanks</p> <p>No it does not. You can click on the links for each build to see which fixes where included in that build at <a href="http://build.whamcloud.com/job/lustre-b2_5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/</a></p> <p>Hit it on master:</p> <p><a href="https://maloo.whamcloud.com/test_sets/a22c319e-8e7f-11e3-b036-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a22c319e-8e7f-11e3-b036-52540035b04c</a></p> <p>Unfortunately, the information about what patches are included in build #6, #7, and #8 is now lost, because Jenkins does not cache build information indefinitely. It would have been useful to paste the list of patches for each build into this bug.</p> <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2)<br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_5/21/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/21/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The same failure occurred, console log on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50h: LU-642: activate deactivated OST ===== 10:06:50 \(1392314810\)^M Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ===== 10:06:50 (1392314810)^M Lustre: DEBUG MARKER: mkdir -p /mnt/mds1^M Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1^M Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/mds1^M LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: ^M Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs.^M Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust^M Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null^M Lustre: MGS: Regenerating lustre-OST0000 log by user request.^M Lustre: Permanently deactivating lustre-OST0000^M Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request^M LustreError: 19638:0:(import.c:284:ptlrpc_invalidate_import()) ASSERTION( imp-&gt;imp_invalid ) failed: ^M LustreError: 19638:0:(import.c:284:ptlrpc_invalidate_import()) LBUG^M Pid: 19638, comm: llog_process_th^M ^M Call Trace:^M [&lt;ffffffffa03f7895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa03f7e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M [&lt;ffffffffa08bc227&gt;] ptlrpc_invalidate_import+0x377/0x880 [ptlrpc]^M [&lt;ffffffffa04077b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs]^M [&lt;ffffffffa0685559&gt;] ? lprocfs_write_frac_helper+0xa9/0x150 [obdclass]^M [&lt;ffffffffa08990f8&gt;] ptlrpc_set_import_active+0x3d8/0x670 [ptlrpc]^M [&lt;ffffffffa1307f0e&gt;] osp_wr_active+0xfe/0x120 [osp]^M [&lt;ffffffffa06a368a&gt;] class_process_proc_param+0x34a/0x770 [obdclass]^M [&lt;ffffffffa12f6b45&gt;] osp_process_config+0x2a5/0x640 [osp]^M [&lt;ffffffffa06a7570&gt;] class_process_config+0x1910/0x1ad0 [obdclass]^M [&lt;ffffffffa06a0362&gt;] ? lustre_cfg_new+0x312/0x6e0 [obdclass]^M [&lt;ffffffffa06a884b&gt;] class_config_llog_handler+0xa2b/0x1880 [obdclass]^M [&lt;ffffffffa04077b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs]^M [&lt;ffffffffa066adab&gt;] llog_process_thread+0x8ab/0xd60 [obdclass]^M [&lt;ffffffffa06b344f&gt;] ? keys_fill+0x6f/0x190 [obdclass]^M [&lt;ffffffffa066b2a5&gt;] llog_process_thread_daemonize+0x45/0x70 [obdclass]^M [&lt;ffffffffa066b260&gt;] ? llog_process_thread_daemonize+0x0/0x70 [obdclass]^M [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0^M [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20^M [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0^M [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20^M ^M Kernel panic - not syncing: LBUG^M </pre> </div></div> <p>Maloo report:<br/> <a href="https://maloo.whamcloud.com/test_sets/0d342454-95b7-11e3-bb07-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0d342454-95b7-11e3-bb07-52540035b04c</a></p> <p>I just cherry-picked the patch to Lustre b2_5 branch (using the "Cherry Pick To" option in Gerrit): <a href="http://review.whamcloud.com/9283" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9283</a></p> <p>Landed for 2.6</p> Duplicate LU-4636 Development Rank 1|hzwbdz: Rank (Obsolete) 12021 Severity