https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3798 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/ae12d972-092d-11e3-b004-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/ae12d972-092d-11e3-b004-52540035b04c</a>.</p> <p>The sub-test test_86 failed with the following error:</p> <blockquote> <p>test_86 failed with 5</p></blockquote> <p>Info required for matching: replay-single 86</p> <p>Client console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:47:43:if [ $running -eq 0 ] ; then 02:47:43: mkdir -p /mnt/lustre; 02:47:43: mount -t lustre -o user_xattr,acl,flock wtm-10vm7@tcp:/lustre /mnt/lustre; 02:47:43: rc=$?; 02:47:43:fi; 02:47:43:exit $rc 02:47:43:LustreError: 152-6: Ignoring deprecated mount option 'acl'. 02:47:43:LustreError: 15c-8: MGC10.10.16.120@tcp: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 02:47:43:LustreError: 9196:0:(llite_lib.c:1046:ll_fill_super()) Unable to process log: -5 02:47:43:Lustre: Unmounted lustre-client </pre> </div></div> LU-3798

replay-single test_86: configuration log errors

Bug Major Resolved Cannot Reproduce Bob Glossman Maloo zfs Tue, 20 Aug 2013 21:17:53 +0000 Tue, 14 Dec 2021 22:52:26 +0000 Tue, 14 Dec 2021 22:52:26 +0000 Lustre 2.5.0 Lustre 2.8.0 0 8 <p>I suspect this is some kind of race between mounting the MDS/MGS (which is slow for some reason) and remounting the client but the MGS is not yet ready. We could have the client retry the mount a couple of times (with "-o retry=5" mount option on the client) to see if that solves the problem?</p> <p>It looks like this test only started failing regularly on July 31st (only 4 failures ever before then, maybe once every few months), and has failed fairly consistently since then (40 failures in 21 days), so it is almost certainly a regression landed on 2013-07-31.</p> <p>The patch in <a href="https://jira.whamcloud.com/browse/LU-3155" title="Permanent parameters with lctl set_param -P" class="issue-link" data-issue-key="LU-3155"><del>LU-3155</del></a> <a href="http://review.whamcloud.com/6025" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6025</a> would be a prime culprit, since it is one of the major changes to configuration that was landed at that time. It would be worthwhile to look through the test failures and find the latest commit that is common to all of them.</p> <p>The below query finds all of the test_86 failures, then each one needs to go to the main "replay-single" test log, then the specific git commit hash for that test needs to be used to find the "parent" on which the patch was based. The latest common parent among all the failures (excepting possibly the failing patch itself) is the likely source of the regression.</p> <p><a href="https://maloo.whamcloud.com/sub_tests/query?commit=Update+results&amp;page=2&amp;sub_test[query_bugs]=&amp;sub_test[status]=FAIL&amp;sub_test[sub_test_script_id]=fcadf0d2-32c3-11e0-a61c-52540025f9ae&amp;test_node[architecture_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;test_node[os_type_id]=&amp;test_node_network[network_type_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=2419200&amp;test_session[test_group]=&amp;test_session[test_host]=&amp;test_session[user_id]=&amp;test_set[test_set_script_id]=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;utf8=%E2%9C%93" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/query?commit=Update+results&amp;page=2&amp;sub_test[query_bugs]=&amp;sub_test[status]=FAIL&amp;sub_test[sub_test_script_id]=fcadf0d2-32c3-11e0-a61c-52540025f9ae&amp;test_node[architecture_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;test_node[os_type_id]=&amp;test_node_network[network_type_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=2419200&amp;test_session[test_group]=&amp;test_session[test_host]=&amp;test_session[user_id]=&amp;test_set[test_set_script_id]=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;utf8=%E2%9C%93</a></p> <p>Bob</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>A couple of key features seen in all the instances found:<br/> 1) only seen in review-zfs test runs, not any in review<br/> 2) test_86 failure isn't the first failed test in replay-single. Always follows other failures, not always the same ones. subtests in test_58 and test_85 are most common.</p> <p>based on 1) we might want to get a zfs expert looking at this.</p> <p>I am continuing to try to narrow it down.</p> <p>Another fact; haven't been any instances seen at all since 8/19. wondering if the problem may have been fixed by a more recent commit.</p> <p>Surveying all the instances the latest common parent is the commit for <a href="https://jira.whamcloud.com/browse/LU-3155" title="Permanent parameters with lctl set_param -P" class="issue-link" data-issue-key="LU-3155"><del>LU-3155</del></a>. That is the same one Andreas called out as suspicious in his comment. Evidence is stacking up that it's the cause.</p> <p>Since evidence suggests <a href="https://jira.whamcloud.com/browse/LU-3155" title="Permanent parameters with lctl set_param -P" class="issue-link" data-issue-key="LU-3155"><del>LU-3155</del></a> was the cause of this problem we need the author to comment.</p> <p>There is now at least one counterexample to the connection of this bug to zfs. At least 1 recent failure was seen in a review run: <a href="https://maloo.whamcloud.com/test_sets/5df51836-132a-11e3-8c44-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5df51836-132a-11e3-8c44-52540035b04c</a></p> <p>Still happens a lot more with zfs.</p> <p>from replay-single.test_86.debug_log.client-26vm2.1378029549 :</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000100:02020000:0.0:1378029519.217418:0:22466:0:(client.c:1168:ptlrpc_check_status()) 11-0: MGC10.10.4.154@tcp: Communicating with 10.10.4.154@tcp, operation ldlm_enqueue failed with -107. .. 10000000:01000000:0.0:1378029519.217530:0:22466:0:(mgc_request.c:1849:mgc_process_log()) Can't get cfg lock: -107 10000000:01000000:0.0:1378029519.217535:0:22466:0:(mgc_request.c:1868:mgc_process_log()) MGC10.10.4.154@tcp: configuration from log <span class="code-quote">'lustre-client'</span> failed (-5). </pre> </div></div> <p>The test is called "replay-single test 86: umount server after clear nid_stats should not hit LBUG" so it looks like we unmount server before config file is processed.</p> <p>hit this error in interop testing between 2.7.0 server and master RHEL6.6 client:<br/> <a href="https://testing.hpdd.intel.com/test_sets/bbeec3b0-454b-11e5-a64b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bbeec3b0-454b-11e5-a64b-5254006e85c2</a></p> <p>another instance:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/3d5c7066-5157-11e5-9f68-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3d5c7066-5157-11e5-9f68-5254006e85c2</a></p> Duplicate LU-3800 Related LU-3155 Development Rank 1|hzvypr: Rank (Obsolete) 9816 Severity