https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15571 Lustre <p>The addition of <tt>iotrace</tt> to the default debug mask is causing interop testing failures with new clients against older servers (eg. 2.14.0) for subtests that restore the debug mask at the end of the test. For example, sanity test_24v:<br/> <a href="https://testing.whamcloud.com/test_sets/96c360e4-cbca-46b4-8a8e-0371f9a8f4b4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/96c360e4-cbca-46b4-8a8e-0371f9a8f4b4</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>onyx-61vm3: error: set_param: setting /sys/kernel/debug/lnet/debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout: Invalid argument pdsh@onyx-61vm1: onyx-61vm3: ssh exited with exit code 22 </pre> </div></div> <p>and on the console logs it shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cfs_str2mask()) unknown mask 'iotrace'. </pre> </div></div> <p>This is likely caused by the test-framework using the client debug mask (which contains iotrace by default) being used on all of the remote nodes. </p> <p>It probably is enough to filter out the "<tt>iotrace</tt>" string from the saved debug mask before using it on the remote node, if the server version is older than 2.14.57 (or whatever version the patch was included in).</p> LU-15571

iotrace debug mask causing interop testing failures

Bug Major Resolved Fixed Patrick Farrell Andreas Dilger Sun, 20 Feb 2022 16:38:16 +0000 Sun, 27 Mar 2022 16:19:55 +0000 Sun, 27 Mar 2022 16:19:55 +0000 Lustre 2.15.0 Lustre 2.15.0 0 6 <p>Tests that are affected include at least sanity 24v, 24A, 27U, but may be more.</p> <p>I'm not sure if there is a 2.15.0 release tracker that may show the other interop issues, but if so then this should be linked there. There do appear to be at least some sanity-flr interop failures that appear functional rather than just test issues. </p> <p>You said adding it to the default debug mask - I don't think we did <b>that</b>, did we?  Just its existence in the debug mask at all, right?</p> <p>It may not be in the <b>default</b> debug mask, but iotrace is at least always enabled when these tests are run. The client saves the output only from "<tt>lctl get_param debug</tt>" and assumes it is the same across the clients and servers, then "restores" it to the older servers, where the error is hit. It looks like this is happening in:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # wrappers <span class="code-keyword">for</span> createmany and unlinkmany # to set debug=0 <span class="code-keyword">if</span> number of creates is high enough # <span class="code-keyword">this</span> is to speedup testing function createmany() { local count=${!#} (( count &gt; 100 )) &amp;&amp; { local saved_debug=$($LCTL get_param -n debug) local list=$(comma_list $(all_nodes)) do_nodes $list $LCTL set_param -n debug=0 } $LUSTRE/tests/createmany $* local rc=$? (( count &gt; 100 )) &amp;&amp; do_nodes $list <span class="code-quote">"$LCTL set_param -n debug=\\\"</span>$saved_debug\\\"" <span class="code-keyword">return</span> $rc } </pre> </div></div> <p>I'll push a patch shortly.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46636" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46636</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15571" title="iotrace debug mask causing interop testing failures" class="issue-link" data-issue-key="LU-15571"><del>LU-15571</del></a> tests: save/restore debug mask for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 2<br/> Commit: 183e00aa0c2892d977f45cda67d6f3352de3429e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46636/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46636/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15571" title="iotrace debug mask causing interop testing failures" class="issue-link" data-issue-key="LU-15571"><del>LU-15571</del></a> tests: save/restore debug mask for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f236119e6e264b00c20533336303f694d9cfe766</p> <p>Landed for 2.15</p> Duplicate LU-15696 Related LU-15317 Development Rank 1|i02itz: Rank (Obsolete) 9223372036854775807 Severity