https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5218 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/c22bdcb4-f4ba-11e3-ae09-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/c22bdcb4-f4ba-11e3-ae09-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>MDS console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:05:56:Lustre: DEBUG MARKER: == lustre-rsync-test test 1: Simple Replication == 19:05:44 (1402797944) 19:05:56:Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_register -n 19:05:56:Lustre: lustre-MDD0000: changelog on 19:05:56:Lustre: DEBUG MARKER: lctl get_param -n mdd.lustre-MDT0000.changelog_users 19:05:56:Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2&gt;&amp;1 | 19:05:56: grep -E -q '(ea_inode|large_xattr)' 19:05:56:Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2&gt;&amp;1 19:05:56:Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2&gt;&amp;1 | 19:05:56: grep -E -q '(ea_inode|large_xattr)' 19:05:56:LustreError: 8689:0:(lustre_log.h:440:llog_group_get_ctxt()) ASSERTION( index &gt;= 0 &amp;&amp; index &lt; LLOG_MAX_CTXTS ) failed: 19:05:56:LustreError: 8689:0:(lustre_log.h:440:llog_group_get_ctxt()) LBUG 19:05:56:Pid: 8689, comm: mdt00_000 19:05:56: 19:05:56:Call Trace: 19:05:56: [&lt;ffffffffa048e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 19:05:56: [&lt;ffffffffa048ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 19:05:56: [&lt;ffffffffa0849868&gt;] llog_origin_handle_open+0x668/0x670 [ptlrpc] 19:05:56: [&lt;ffffffffa088db35&gt;] tgt_llog_open+0x35/0xd0 [ptlrpc] 19:05:56: [&lt;ffffffffa08942cc&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] 19:05:56: [&lt;ffffffffa0843d3a&gt;] ptlrpc_main+0xd1a/0x1980 [ptlrpc] 19:05:56: [&lt;ffffffffa0843020&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] 19:05:56: [&lt;ffffffff8109ab56&gt;] kthread+0x96/0xa0 19:05:56: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 19:05:56: [&lt;ffffffff8109aac0&gt;] ? kthread+0x0/0xa0 19:05:56: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 19:05:56: 19:05:56:Kernel panic - not syncing: LBUG 19:05:56:Pid: 8689, comm: mdt00_000 Not tainted 2.6.32-431.17.1.el6_lustre.g8d5344f.x86_64 #1 19:05:56:Call Trace: 19:05:56: [&lt;ffffffff8152795f&gt;] ? panic+0xa7/0x16f 19:05:56: [&lt;ffffffffa048eeeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 19:05:56: [&lt;ffffffffa0849868&gt;] ? llog_origin_handle_open+0x668/0x670 [ptlrpc] 19:05:56: [&lt;ffffffffa088db35&gt;] ? tgt_llog_open+0x35/0xd0 [ptlrpc] 19:05:56: [&lt;ffffffffa08942cc&gt;] ? tgt_request_handle+0x23c/0xac0 [ptlrpc] 19:05:56: [&lt;ffffffffa0843d3a&gt;] ? ptlrpc_main+0xd1a/0x1980 [ptlrpc] 19:05:56: [&lt;ffffffffa0843020&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] 19:05:56: [&lt;ffffffff8109ab56&gt;] ? kthread+0x96/0xa0 19:05:56: [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 19:05:56: [&lt;ffffffff8109aac0&gt;] ? kthread+0x0/0xa0 19:05:56: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 19:05:56:Initializing cgroup subsys cpuset 19:05:56:Initializing cgroup subsys cpu 19:05:56:Linux version 2.6.32-431.17.1.el6_lustre.g8d5344f.x86_64 (jenkins@builder-2- </pre> </div></div> server: lustre-master build # 2901<br/> client: 2.5.1 LU-5218

Interop 2.5.1<->2.6 failure on test suite lustre-rsync-test test_1: ASSERTION( index >= 0 && index < LLOG_MAX_CTXTS ) failed

Bug Blocker Resolved Fixed Mikhail Pershin Maloo HB Wed, 18 Jun 2014 00:58:02 +0000 Wed, 25 Jun 2014 17:44:12 +0000 Wed, 25 Jun 2014 17:43:27 +0000 Lustre 2.6.0 Lustre 2.6.0 0 6 <p>With new crashdumps feature now available for maloo runs, please also check that this crashdump is available and also use it to see what index value is I guess.</p> <p>this is related to commit 14d162c5438de959d0ea01fb1b40a7c5dfa764d1</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">@@ -208,15 +204,9 @@ <span class="code-keyword">enum</span> llog_ctxt_id { LLOG_CONFIG_ORIG_CTXT = 0, LLOG_CONFIG_REPL_CTXT, LLOG_MDS_OST_ORIG_CTXT, - LLOG_MDS_OST_REPL_CTXT, LLOG_SIZE_ORIG_CTXT, LLOG_SIZE_REPL_CTXT, - LLOG_RD1_ORIG_CTXT, - LLOG_RD1_REPL_CTXT, LLOG_TEST_ORIG_CTXT, - LLOG_TEST_REPL_CTXT, - LLOG_LOVEA_ORIG_CTXT, - LLOG_LOVEA_REPL_CTXT, LLOG_CHANGELOG_ORIG_CTXT, <span class="code-comment">/**&lt; changelog generation on mdd */</span> LLOG_CHANGELOG_REPL_CTXT, <span class="code-comment">/**&lt; changelog access on clients */</span> LLOG_CHANGELOG_USER_ORIG_CTXT, <span class="code-comment">/**&lt; <span class="code-keyword">for</span> multiple changelog consumers */</span> </pre> </div></div> <p>some entries were removed from enum, so all llog context names have new values since that moment and incompatible with all pre-commit Lustre versions. This is critical for change log and HSM agents in old versions. I am not sure what would be better fix - to assign old numbers to the entries or to have two tables and use one or another depending on connected client version.</p> <p>the patch is here - <a href="http://review.whamcloud.com/10758" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10758</a></p> <p>I've just reverted things back, add comments and check for context index from wire. </p> <p>What a gigantic mess. It surprises me that these <tt>llog_ctxt_id</tt> values have been in the network protocol since ancient times, but they are not declared in lustre_idl.h, and are not checked in wirecheck/wiretest. What is the point of calling <tt>llog_open(..., name = CHANGELOG_CATALOG, ...)</tt> if it is passing <tt>body-&gt;lgd_ctxt_idx = LLOG_CHANGELOG_REPL_CTXT</tt> and using that to open the log?</p> <p>For the 2.6 release I think we don't have any option except to move <tt>enum llog_ctxt_idx</tt> into lustre_idl.h and assign the specific (old) values to each of the remaining <tt>LLOG_.*&#95;CTXT</tt> values. They should be added to wirecheck/wiretest on b2_5. It looks like in ancient times on HEAD these values were specifically assigned since commit d2d56f38, but these values were removed by a patch from you &amp; Shadow in commit c9842fdc (some time in 1.6 via <a href="https://bugzilla.lustre.org/show_bug.cgi?id=13821" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=13821</a>) and that removed the <tt>LLOG&#95;MD&#95;{ORIG,REPL}&#95;CTXT</tt> values and the explicit assignments, so the resulting values changed. Even worse, the c9842fdc patch changed the values after <tt>LLOG_MD_ORIG_CTXT</tt> to be different on master and b1_8 (where the enum llog_ctxt_idx still has explicit values assigned). Fortunately, b1_8 does not have any HSM support, and we do not use <tt>LLOG_TEST_{ORIG,REPL}&#95;CTXT</tt> over the network so the fact that the assigned values at the high end are different does not impact anything.</p> <p>I think the correct assignments today should be:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">enum</span> llog_ctxt_id { LLOG_CONFIG_ORIG_CTXT = 0, LLOG_CONFIG_REPL_CTXT = 1, LLOG_MDS_OST_ORIG_CTXT = 2, LLOG_MDS_OST_REPL_CTXT = 3, LLOG_SIZE_ORIG_CTXT = 4, LLOG_SIZE_REPL_CTXT = 5, LLOG_TEST_ORIG_CTXT = 8, LLOG_TEST_REPL_CTXT = 9, LLOG_CHANGELOG_ORIG_CTXT = 12, <span class="code-comment">/**&lt; changelog generation on mdd */</span> LLOG_CHANGELOG_REPL_CTXT = 13, <span class="code-comment">/**&lt; changelog access on clients */</span> LLOG_CHANGELOG_USER_ORIG_CTXT = 14, <span class="code-comment">/**&lt; <span class="code-keyword">for</span> multiple changelog consumers */</span> LLOG_AGENT_ORIG_CTXT = 15, <span class="code-comment">/**&lt; agent requests generation on cdt */</span> LLOG_MAX_CTXTS } </pre> </div></div> <p>There should be a big comment explaining this and referencing this LU ticket. The LASSERT() should be avoided by <tt>llog_origin_handle_open()</tt> checking the value of <tt>body-&gt;lgh_ctxt_idx</tt> is valid before calling <tt>llog_get_context()</tt>, and return <tt>-EPROTO</tt> if it is bad.</p> <p>Moved this to be a blocker because without this patch HSM interoperability between the MDS and archive clients will be broken.</p> <p>In addition to the proposed patch - we need another patch to fix the assertion on network data, I think.</p> <p>Oleg, what assertion do you mean?</p> <p>The assertion that was reported initially here, about hte index being out of range.</p> <p>Patch landed to Master.</p> <p>The original LASSERT() cannot be hit with the patch that was landed. The code checks that the requested context is valid before calling down into the code. </p> Duplicate LU-5257 Related LU-5230 Development Rank 1|hzwpaf: Rank (Obsolete) 14552 Severity