https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11058 Lustre <p>This issue was created by maloo for John Hammond &lt;john.hammond@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/4155343e-5de1-11e8-abc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4155343e-5de1-11e8-abc3-52540065bddc</a></p> <p>test_77k failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>checksum(1) != 0 </pre> </div></div> <p>This subtest was added by the recently landed change <a href="https://review.whamcloud.com/32095" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32095</a> <a href="https://jira.whamcloud.com/browse/LU-10906" title="checksums parameter not persistent after reboot" class="issue-link" data-issue-key="LU-10906"><del>LU-10906</del></a> checksum: enable/disable checksum correctly.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_77k - checksum(1) != 0</p> LU-11058

sanity test_77k: checksum(1) != 0

Bug Major Reopened Unresolved Hongchao Zhang Maloo always_except Fri, 25 May 2018 14:22:23 +0000 Fri, 2 Aug 2019 19:11:48 +0000 Lustre 2.12.0 0 14 <p>Emoly, it looks like your previous patch broke sanity.sh test_77k.</p> <p>This test failed about 30 times so far.</p> <p>I will look into it.</p> <p>I suspect the failure may be related to the processing speed of llog thread. Although it is called before client_common_fill_super(), but is it possible to set checksum by set_param later than obd_set_info_async?</p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b1aa5138-65b6-11e8-907b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b1aa5138-65b6-11e8-907b-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e3174aec-65ba-11e8-907b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e3174aec-65ba-11e8-907b-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/a7dc29fe-65c7-11e8-aa24-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a7dc29fe-65c7-11e8-aa24-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/36eefcd2-65cf-11e8-a55d-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/36eefcd2-65cf-11e8-a55d-52540065bddc</a></p> <p>test 77k is failing a lot &amp; it's only been there a few days</p> <p>Error: 'checksum(1) != 0' <br/> Failure Rate: 11.00% of most recent 100 runs, 0 skipped (all branches)</p> <p>Please submit a patch to move this test into <tt>ALWAYS_EXCEPT</tt> for now. . </p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32685" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32685</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> tests: stop running sanity test 77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b9a2bd475acdeecc3cb372c2e5b5c530979f8f2b</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="https://review.whamcloud.com/32685/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32685/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> tests: stop running sanity test 77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3ddee05e6106d3336081cc2d6d7b681116263829</p> <p>This bug is not fixed with this landing, do not close it. </p> <p>I looked at this a bit. When it works the params log is processed normally. When it doesn't work <tt>mgc_process_log()</tt> skips the params log since <tt>cld-&gt;cld_stopping</tt> is set. So maybe due finding an old CLD of the params log.</p> <p>I can easily produce this issue on ldiskfs locally. The log shows it's related to the following code in process_param2_config():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (kobj) { char *value = param; char *envp[3]; int i; param = strsep(&amp;value, "="); envp[0] = kasprintf(GFP_KERNEL, "PARAM=%s", param); envp[1] = kasprintf(GFP_KERNEL, "SETTING=%s", value); envp[2] = NULL; rc = kobject_uevent_env(kobj, KOBJ_CHANGE, envp); for (i = 0; i &lt; ARRAY_SIZE(envp); i++) kfree(envp[i]); kobject_put(kobj); RETURN(rc); } </pre> </div></div> <p>What shows up in the log exactly? It is just sending a uevent. Is the memory allocation failing? This code also only gets executed with lctl set_param -P. Are you running with that command? Alrighty let me take a look.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33099" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33099</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> test: reenable sanity 77k test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 38041bef21baaec1486c3e31f14f7aaeee1ba6bf</p> <p>I can't seem to reproduce the issue in my test bed. I pushed a patch so I can collect debug logs. Assuming it still fails <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>If you run:</p> <p>udevadm monitor --subsystem-match=lustre --property</p> <p>You should see the uevents being sent and what values.</p> <p>I bet you are testing from the lustre source tree. In that case you lctl is missing in /usr/sbin and the uevent will be missed. If you make install into your image or copy 99-lustre.rules to /etc/udev/rules.d/ then it will go further.</p> <p>Yes, I'm using lustre source tree for test. I will try per your comment and see. Thanks.</p> <p>Just to repeat my comment, I think this is due to a llog issue. (Which may be exposed in some ray by recent changes.)</p> <p>I looked at this a bit. When it works the params log is processed normally. When it doesn't work mgc_process_log() skips the params log since cld-&gt;cld_stopping is set. So I think this is due finding an old CLD of the params log.</p> <p>The patch <a href="https://review.whamcloud.com/#/c/33190" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33190</a> seems to have resolved this bug</p> <p>ok, let's mark it a duplicate of <a href="https://jira.whamcloud.com/browse/LU-10595" title="Use after free in mgc_process_cfg_log" class="issue-link" data-issue-key="LU-10595"><del>LU-10595</del></a> then</p> <p>Reopen until test is removed from <tt>ALWAYS_EXCEPT</tt>.</p> <p>Please remove <tt>always_except</tt> label when that patch is landed.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33099/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33099/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> test: reenable sanity 77k test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fb1875c314cf3650aa7ba6b2cdb72ef18e812d5f</p> <p>Multiple failures of test_77k very recently, for instance:<br/> <a href="https://testing.whamcloud.com/test_sets/376fee56-cc76-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/376fee56-cc76-11e8-ad90-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/e992b980-cc92-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e992b980-cc92-11e8-ad90-52540065bddc</a></p> <p>Since removing this test from the ALWAYS_EXCEPT list and since landing the patch for <a href="https://jira.whamcloud.com/browse/LU-10595" title="Use after free in mgc_process_cfg_log" class="issue-link" data-issue-key="LU-10595"><del>LU-10595</del></a> <a href="https://review.whamcloud.com/#/c/33190" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33190</a>, sanity test 77k has failed with this error three times; the two failures Sebastien listed above and <a href="https://testing.whamcloud.com/test_sets/c1ba7a5a-cc7f-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c1ba7a5a-cc7f-11e8-ad90-52540065bddc</a> .</p> <p>So far, the three failures are in review-ldiskfs.</p> <p>I think the test is wrong with how its using lctl set_param '-P'. Now that the <a href="https://jira.whamcloud.com/browse/LU-7004" title="fix &quot;lctl set_param -P&quot; to allow deprecation of &quot;lctl conf_param&quot;" class="issue-link" data-issue-key="LU-7004"><del>LU-7004</del></a> test suite patch landed I think we can move this test to the new function and it will work.</p> <p>Working on patch</p> <p>Okay I moved to the test to the new set_persistent_param_and_check() which allows us to test both lctl set_param -P and lctl conf_param. I'm seeing failures with the lctl conf_param case. It will take a few hours to sort this out.</p> <p>So from my understanding the mount option <span class="error">&#91;no&#93;</span>checksum over rides all other checksum tuning. For example mount --nochecksum should prevent lctl set_param -P / lctl conf_param from taking affect. This should be true of the direct sysfs tunings as well? Is this thinking correct?</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33353" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33353</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> tests: stop running sanity test 77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f78e8ff46acb649ecb76607fe13a227a70c28c62</p> <p>sanity test 77k is failing about 5% of the time for review-ldiskfs, review-dne and review-dne-zfs. </p> <p>I prepared a patch to stop running sanity test 77k in case this is too disruptive to patch landings.</p> <p>I think I have a proper fix. I'm currently testing it.</p> <p>Oleg Drokin (green@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33354" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33354</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> test: reenable sanity 77k test"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 179b1a83e6ce4eb2310fef67b23dd16b7c32a9b1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33354/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33354/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> test: reenable sanity 77k test"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4839674385f2f2cf25206ea24dd3999096c29d59</p> <p>James </p> <p>we're reverting the test being on for now so please turn it back on along with your fix when it is ready</p> <p>Thanks</p> <p>Peter</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33363" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33363</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> obd: manage checksum state<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0bb53a3db196d82df918f75f653859ed78a903f8</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34035" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34035</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> tests: cleanup persistent checksum= in 77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5a85e3040d7da834b17e311b834b3aed202d6cd4</p> <p>Note the test fix is only a work around. The proper is still to be worked out.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34035/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34035/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11058" title="sanity test_77k: checksum(1) != 0" class="issue-link" data-issue-key="LU-11058">LU-11058</a> tests: cleanup persistent checksum= in 77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 65a7a268251634acd81dbf14dc78f1b4f7cf5157</p> Related LU-5020 LU-10595 LU-10906 LU-11263 LU-9398 Development Rank 1|hzzxtz: Rank (Obsolete) 9223372036854775807 Severity