Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-5361] sanity test_77i: algo set to crc32 instead of adler https://jira.whamcloud.com/browse/LU-5361 Lustre <p>This issue was created by maloo for John Hammond <john.hammond@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/307abe62-0cd8-11e4-8ce3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/307abe62-0cd8-11e4-8ce3-5254006e85c2</a>.</p> <p>The sub-test test_77i failed with the following error:</p> <blockquote> <p>algo set to crc32 instead of adler</p></blockquote> <p>Info required for matching: sanity 77i</p> LU-5361

sanity test_77i: algo set to crc32 instead of adler

Bug Minor Resolved Fixed WC Triage Maloo Wed, 16 Jul 2014 14:30:56 +0000 Tue, 22 May 2018 06:01:13 +0000 Tue, 14 Mar 2017 06:05:18 +0000 Lustre 2.9.0 Lustre 2.10.0 0 14 <p>Seems like a bogus test to me. There's no guarantee that adler will be faster than crc32 all the time.</p> <p>Completely agree. On systems with hardware accelerated crc32 it will beat alder32 hands down.</p> <p>Note also that we have recently changed the crypto speed test code. See <a href="https://jira.whamcloud.com/browse/LU-5063" title="sanity-quota test_7b test_12a test_13: dd failed" class="issue-link" data-issue-key="LU-5063">LU-5063</a> and <a href="https://jira.whamcloud.com/browse/LU-5279" title="kernel BUG at arch/x86/mm/physaddr.c:20!" class="issue-link" data-issue-key="LU-5279"><del>LU-5279</del></a>.</p> <p>When the test was originally written, the "adler" algorithm was always available since the availability had nothing to do with performance. The change to make only a subset of algorithms available to the client was done in 2.2.58 so that clients would not choose the locally-fastest algorithm at the expense of selecting a slow algorithm on the server.</p> <p>That said, this test should be exercising the client checksum selection if OBD_CONNECT_CKSUM is not available, in which case it should select adler:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_77i() { # bug 13805 #define OBD_FAIL_OSC_CONNECT_CKSUM 0x40b lctl set_param fail_loc=0x40b remount_client $MOUNT lctl set_param fail_loc=0 <span class="code-keyword">for</span> VALUE in `lctl get_param osc.*osc-[^mM]*.checksum_type`; <span class="code-keyword">do</span> PARAM=`echo ${VALUE[0]} | cut -d <span class="code-quote">"="</span> -f1` algo=`lctl get_param -n $PARAM | sed <span class="code-quote">'s/.*\[\(.*\)\].*/\1/g'</span>` [ <span class="code-quote">"$algo"</span> = <span class="code-quote">"adler"</span> ] || error <span class="code-quote">"algo set to $algo instead of adler"</span> done remount_client $MOUNT } run_test 77i <span class="code-quote">"client not supporting OSD_CONNECT_CKSUM"</span> </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (!OBD_FAIL_CHECK(OBD_FAIL_OSC_CONNECT_CKSUM)) { /* OBD_CONNECT_CKSUM should always be set, even <span class="code-keyword">if</span> checksums are * disabled by <span class="code-keyword">default</span>, because it can still be enabled on the * fly via /proc. As a consequence, we still need to come to an * agreement on the supported algorithms at connect time */ data->ocd_connect_flags |= OBD_CONNECT_CKSUM; <span class="code-keyword">if</span> (OBD_FAIL_CHECK(OBD_FAIL_OSC_CKSUM_ADLER_ONLY)) data->ocd_cksum_types = OBD_CKSUM_ADLER; <span class="code-keyword">else</span> data->ocd_cksum_types = cksum_types_supported_client(); } </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ocd->ocd_connect_flags & OBD_CONNECT_CKSUM) { /* We sent to the server ocd_cksum_types with bits set * <span class="code-keyword">for</span> algorithms we understand. The server masked off * the checksum types it doesn't support */ : : } <span class="code-keyword">else</span> { /* The server does not support OBD_CONNECT_CKSUM. * Enforce ADLER <span class="code-keyword">for</span> backward compatibility*/ cli->cl_supp_cksum_types = OBD_CKSUM_ADLER; } </pre> </div></div> <p>though the old comments and debug message in <tt>ofd_parse_connect_data()</tt> would have you think otherwise (they were not updated as part of <a href="http://review.whamcloud.com/3098" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3098</a> when the fallback was changed from crc32 to adler), but that should not actually have any effect on operation:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } <span class="code-keyword">else</span> { /* This client does not support OBD_CONNECT_CKSUM * fall back to CRC32 */ CDEBUG(D_RPCTRACE, <span class="code-quote">"%s: cli %s does not support "</span> <span class="code-quote">"OBD_CONNECT_CKSUM, CRC32 will be used\n"</span>, exp->exp_obd->obd_name, obd_export_nid2str(exp)); } </pre> </div></div> <p>So this test shouldn't have anything to do with the relative speed of the algorithms.</p> <p>It may be that the test is racy and the fail_loc is cleared before all of the OSC connections have been made? It might make sense to move the <tt>fail_loc=0</tt> line to just before the second remount to avoid this problem.</p> <p>When fixing this, the test_77i() code could be cleaned up a bit. It uses <tt>cut -d=</tt> when it could be using <tt>lctl get_param -N</tt> to just print the parameter names. It should also use $(...) for subshells instead of `...`, and it should print out the OSC device on which the failure(s) happened.</p> <p>Actually, the whole check could be simplified to:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl get_param osc.*osc-[^mM]*.checksum_type | grep -v <span class="code-quote">'\[adler\]'</span> && error <span class="code-quote">"checksum algorithm not set to adler"</span> </pre> </div></div> <p>Note also that this test failure appears to have been around a long time and is unlikely to be (directly) related to the checksum algorithm changes. <a href="https://jira.whamcloud.com/browse/LU-3243" title="sanity.sh test_77i: algo set to crc32 instead of adler" class="issue-link" data-issue-key="LU-3243"><del>LU-3243</del></a> predates it, and was closed when the debug patch was landed, even though the comment said:</p> <blockquote> <p>I submitted <a href="http://review.whamcloud.com/6205" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6205</a> to improve the error message, but that is not fixing the bug here.</p></blockquote> <p>I also see several cases of this test failing if there is an error reading from /proc, but I think those predate the 6205 patch, which prints out the actual checksum algorithm used and doesn't just fail if '<span class="error">[adler]</span>' is not found.</p> <p>Another instance of this error:<br/> 2015-07-16 03:17:56 - <a href="https://testing.hpdd.intel.com/test_sets/156e74ae-2b83-11e5-b0b2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/156e74ae-2b83-11e5-b0b2-5254006e85c2</a><br/> 2015-07-20 15:59:05 - <a href="https://testing.hpdd.intel.com/test_sets/03de714c-2f13-11e5-bc70-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/03de714c-2f13-11e5-bc70-5254006e85c2</a><br/> 2015-08-21 10:53:05 - <a href="https://testing.hpdd.intel.com/test_sets/7461086c-483b-11e5-813b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7461086c-483b-11e5-813b-5254006e85c2</a></p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/269d217c-738a-11e5-ae2e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/269d217c-738a-11e5-ae2e-5254006e85c2</a></p> <p>+1 again on master : <a href="https://testing.hpdd.intel.com/test_sets/aa999490-764d-11e5-86b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/aa999490-764d-11e5-86b7-5254006e85c2</a></p> <p>Another failure on master at <a href="https://testing.hpdd.intel.com/test_sets/1654529c-7e46-11e5-965a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1654529c-7e46-11e5-965a-5254006e85c2</a><br/> 2015-11-13 21:15:03 - <a href="https://testing.hpdd.intel.com/test_sets/aae313cc-8a66-11e5-ba42-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/aae313cc-8a66-11e5-ba42-5254006e85c2</a><br/> 2016-01-21 15:34:52 - <a href="https://testing.hpdd.intel.com/test_sets/70bbba40-c06d-11e5-acc7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/70bbba40-c06d-11e5-acc7-5254006e85c2</a><br/> 2016-01-21 23:10:38 - <a href="https://testing.hpdd.intel.com/test_sets/cb1f99b2-c0ad-11e5-b0fe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cb1f99b2-c0ad-11e5-b0fe-5254006e85c2</a><br/> 2016-02-01 15:26:49 - <a href="https://testing.hpdd.intel.com/test_sets/11b856f0-c911-11e5-bedf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/11b856f0-c911-11e5-bedf-5254006e85c2</a><br/> 2016-02-08 08:55:14 - <a href="https://testing.hpdd.intel.com/test_sets/2c566f30-ce8c-11e5-af15-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2c566f30-ce8c-11e5-af15-5254006e85c2</a><br/> 2016-02-08 16:43:35 - <a href="https://testing.hpdd.intel.com/test_sets/2a62fb4c-ce9d-11e5-bfa1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2a62fb4c-ce9d-11e5-bfa1-5254006e85c2</a></p> <p>Another failure at <a href="https://testing.hpdd.intel.com/test_sets/1654529c-7e46-11e5-965a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1654529c-7e46-11e5-965a-5254006e85c2</a></p> <p>The algorithm used depends on the CPU hardware support. It may be that CRC32 is a lot faster than Adler, so Adler is not available. That probably needs a fix to the test. </p> <p>Ashish Purkar (ashish.purkar@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/23644" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23644</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5361" title="sanity test_77i: algo set to crc32 instead of adler" class="issue-link" data-issue-key="LU-5361"><del>LU-5361</del></a> tests: Incorrect checksum algorithm reported<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a476729a9ca737bd00d611c870837d2b7977317c</p> <p>One more failure instance on master branch:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/a136d632-b36a-11e6-a0b8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a136d632-b36a-11e6-a0b8-5254006e85c2</a></p> <p>Abrarahmed Momin (kais_abrar@yahoo.co.in) uploaded a new patch: <a href="https://review.whamcloud.com/25745" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25745</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5361" title="sanity test_77i: algo set to crc32 instead of adler" class="issue-link" data-issue-key="LU-5361"><del>LU-5361</del></a> llite: Remove OBD_FAIL_OSC_CONNECT_CKSUM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b3e0a5be1edf02b8facb8928b746c23e17a376c3</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23644/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23644/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5361" title="sanity test_77i: algo set to crc32 instead of adler" class="issue-link" data-issue-key="LU-5361"><del>LU-5361</del></a> llite: Remove OBD_FAIL_OSC_CONNECT_CKSUM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5c3151841520001b75e908af0fc3446d80853969</p> <p>Landed for 2.10</p> Related LU-3243 Development Rank 1|hzwrmf: Rank (Obsolete) 14946 Severity