https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16462 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run on sles12.5 clients:<br/> <a href="https://testing.whamcloud.com/test_sets/258bf667-e863-4adb-af68-213f7877b909" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/258bf667-e863-4adb-af68-213f7877b909</a><br/> <a href="https://testing.whamcloud.com/test_sets/e3b187ad-af69-4bd1-b43c-583da240aef3" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e3b187ad-af69-4bd1-b43c-583da240aef3</a></p> <p>test_43a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl dl BUG at file position attr.c:201:validate_nla lctl: attr.c:201: validate_nla: Assertion `0' failed. </pre> </div></div> <p>The same failure exists in a number of other subtests that also use "<tt>lctl dl</tt>":</p> <ul class="alternate" type="square"> <li>sanity: test_33i, test_104d, test_154d</li> <li>conf-sanity: test_43b, test_70c, test_91</li> </ul> <p>It looks like the <tt>validate_nla()</tt> function is part of libnl (netlink), so very likely relates to the new usage of netlink in "<tt>lctl dl</tt>" to get the device list.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> conf-sanity test_70c - set deactivate failed</p> LU-16462

conf-sanity sles12.5 test_43a: lctl: attr.c:201: validate_nla: Assertion `0' failed.

Bug Minor Resolved Fixed James A Simmons Maloo Wed, 11 Jan 2023 23:52:36 +0000 Fri, 21 Jul 2023 04:17:52 +0000 Mon, 1 May 2023 06:27:50 +0000 Lustre 2.16.0 Lustre 2.16.0 0 6 <p>James, can you please take a look.</p> <p>I don't think SLES12 clients are in such heavy usage that they need to get the latest netlink functionality, but at one point at least Cray was heavily based on SLES for their client distro, so at least we shouldn't break it gratuitously. If there isn't a straight forward way to fix it, I'd be fine with just configuring out the netlink functionality and always using ioctl/debugfs in this case (which isn't worse than what was available before).</p> <p>What version of libnl is installed? Do we have a special Test-parameter tag for SUSE12. I suspect that the libnl library is older so its lacking proper support for NLA_S64.</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49608" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49608</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16462" title="conf-sanity sles12.5 test_43a: lctl: attr.c:201: validate_nla: Assertion `0&#39; failed." class="issue-link" data-issue-key="LU-16462"><del>LU-16462</del></a> utils: handle lack of NLA_S64<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a6780268c84f876b98f8d31a6830dcb9acf16077</p> <p>This is preventing patches on e2fsprogs from passing testing, since they run with sles12sp5 clients. </p> <p><a href="https://testing.whamcloud.com/test_sessions/96f8909f-bbaa-40f2-b969-f261d4b0398f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/96f8909f-bbaa-40f2-b969-f261d4b0398f</a><br/> <a href="https://testing.whamcloud.com/test_sessions/1f946f9d-86de-401d-9c45-b3b445eb10b4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/1f946f9d-86de-401d-9c45-b3b445eb10b4</a></p> <p>Comment from Dongyang in the 49608 patch that explains the issue:</p> <blockquote> <p>lctl dl is triggering assert inside libnl3 because we use <tt>NLA_NUL_STRING</tt>.<br/> in the old libnl3, we don't have <tt>NLA_NUL_STRING</tt> and <tt>NLA_S8|16|32|64</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>enum { NLA_UNSPEC, /**&lt; Unspecified type, binary data chunk */ NLA_U8, /**&lt; 8 bit integer */ NLA_U16, /**&lt; 16 bit integer */ NLA_U32, /**&lt; 32 bit integer */ NLA_U64, /**&lt; 64 bit integer */ NLA_STRING, /**&lt; NUL terminated character string */ NLA_FLAG, /**&lt; Flag */ NLA_MSECS, /**&lt; Micro seconds (64bit) */ NLA_NESTED, /**&lt; Nested attributes */ __NLA_TYPE_MAX, }; #define NLA_TYPE_MAX (__NLA_TYPE_MAX - 1) </pre> </div></div> <p>and if we try to use any <tt>NLA_TYPE</tt> greater than <tt>NLA_TYPE_MAX</tt>, it will trigger the assert in <tt>validate_nla()</tt>.<br/> Do we have to use <tt>NLA_NUL_STRING</tt> instead of <tt>NLA_STRING</tt>, and the signed nla types?</p></blockquote> <p>Hi James, any thought on how to make progress on this issue?</p> <p>We have e2fsck fixes blocked from landing for a couple of months because the netlink patch has broken "<tt>lctl dl</tt>" on SLES12 clients. I don't think we need to retroactively add support for SLES12 clients to allow non-root users to run "<tt>lctl dl</tt>", so it would be fine if the netlink code was completely disabled for older clients that don't have <tt>NLA_S32</tt> or <tt>NLA_NUL_STRING</tt> and only the ioctl fallback was used. It just needs to not break the old code.</p> <p>I'm going to push a patch that disables yaml netlink usage if <tt>NLA_NUL_STRING</tt> is not defined. This works fine for "<tt>lctl dl</tt>" in my local testing, but still need to fix "<tt>lctl ping</tt>" (sanity test_217).</p> <p><del>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50610</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-16462" title="conf-sanity sles12.5 test_43a: lctl: attr.c:201: validate_nla: Assertion `0&#39; failed." class="issue-link" data-issue-key="LU-16462"><del>LU-16462</del></a> utils: skip netlink for old libnl3</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 5ff26d3c3205baacefb3c0e48c2a03ff0713db39</del></p> <p>patch <a href="https://review.whamcloud.com/49608" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49608</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16462" title="conf-sanity sles12.5 test_43a: lctl: attr.c:201: validate_nla: Assertion `0&#39; failed." class="issue-link" data-issue-key="LU-16462"><del>LU-16462</del></a> utils: handle lack of NLA_S64</tt>" has been updated to handle the sles12sp5 libnl incompatibility, along with test and tool fixes for the netlink-unavailable fallback case so that "<tt>lctl ping</tt>" and "<tt>lctl list_nids</tt>" continue to work.</p> <p>Thank you Andreas for your help <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I just ran across an old patch from Amir that is replacing usage of "<tt>lctl ping</tt>" and "<tt>lctl list_nids</tt>" with the equivalent "<tt>lnetctl</tt>" commands.</p> <p>The output is clunky and needs some <tt>awk</tt> to parse it into just a NID:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.10.99@tcp status: up interfaces: 0: enp0s3 $ lnetctl net show | awk '/nid:/ &amp;&amp; $3 != "0@lo" { print $3 }' 192.168.10.99@tcp </pre> </div></div> <p>Alexey suggested in that patch to put this into a helper function on <tt>test-framework.sh</tt> instead of having it inline in multiple places. However, users would probably also want to print some of these fields outside of the testing, instead of the full YAML.</p> <p>Having a command-line argument like "<tt>lnetctl net show -<del>nid</tt>" in this case, but also able to print other fields like "<tt>... --status</tt>", "<tt></del><del>nettype</tt>", "<tt></del>-interfaces</tt>") would more convenient than users having to use "<tt>awk</tt>" or "<tt>yq</tt>" to extract the fields manually.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49608/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49608/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16462" title="conf-sanity sles12.5 test_43a: lctl: attr.c:201: validate_nla: Assertion `0&#39; failed." class="issue-link" data-issue-key="LU-16462"><del>LU-16462</del></a> utils: handle lack of newer nla_attrs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ae1ee11cea0a90631e14d670883528d6ac6e86b7</p> <p>Landed for 2.16</p> Related LU-10003 LU-9680 LU-16694 Development Rank 1|i039pb: Rank (Obsolete) 9223372036854775807 Severity