<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 02:51:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-12350] sanity-flr test_33: file content error: expected: ost1, actual: ost2</title>
                <link>https://jira.whamcloud.com/browse/LU-12350</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;This issue was created by maloo for Andreas Dilger  &amp;lt;adilger@whamcloud.com&amp;gt;&lt;/p&gt;

&lt;p&gt;This issue relates to the following test suite run: &lt;a href=&quot;https://testing.whamcloud.com/test_sets/dfadbab6-2668-11e9-a318-52540065bddc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://testing.whamcloud.com/test_sets/dfadbab6-2668-11e9-a318-52540065bddc&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;test_33 failed with the following error:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;&apos;file content error: expected: ost1, actual: ost2&apos;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;First test failure is on 2019-02-01 on &lt;a href=&quot;https://review.whamcloud.com/34160&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch 34160&lt;/a&gt; that didn&apos;t land until 2019-05-24 (so could not have been the cause).  The &lt;a href=&quot;https://testing.whamcloud.com/test_sets/83b5e368-2e52-11e9-9b3a-52540065bddc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;second test failure&lt;/a&gt; is on 2019-02-11 on &lt;a href=&quot;https://review.whamcloud.com/34186&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch 34186&lt;/a&gt; that hasn&apos;t landed as of 2019-05-28, so it must have been a patch landed to master.  Not to be confused with &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10100&quot; title=&quot;sanity test_27a: setstripe failed with &amp;quot;error on ioctl 0x8008669a for &amp;#39;*&amp;#39; (3): Invalid argument&amp;quot;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10100&quot;&gt;&lt;del&gt;LU-10100&lt;/del&gt;&lt;/a&gt;, which is a PPC-specific failure that causes many sanity-flr and other test failures.&lt;/p&gt;

&lt;p&gt;There were a bunch of patches landed on 2019-01-30, but looking through the patch summaries doesn&apos;t show anything that is related.  Since it fails only test_33 about 0.4% of all sanity-flr test runs (about 5x per month), it could have been a patch that landed any time in the previous week or two, but unlikely before that (unless some external environment change contributed to the failure).   The test itself was added in 2017-09-15 so had been passing for a long time.&lt;/p&gt;

&lt;p&gt;My first guess is some kind of a test problem, so dumping &quot;{{lfs getstripe $DIR/&lt;/p&gt;





&lt;p&gt;VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV&lt;br/&gt;
sanity-flr test_33 - &apos;file content error: expected: ost1, actual: ost2&apos;&lt;/p&gt;</description>
                <environment></environment>
        <key id="55774">LU-12350</key>
            <summary>sanity-flr test_33: file content error: expected: ost1, actual: ost2</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.whamcloud.com/images/icons/priorities/minor.svg">Minor</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="pfarrell">Patrick Farrell</assignee>
                                    <reporter username="maloo">Maloo</reporter>
                        <labels>
                    </labels>
                <created>Tue, 28 May 2019 20:25:03 +0000</created>
                <updated>Thu, 20 Jun 2019 22:26:48 +0000</updated>
                            <resolved>Sat, 1 Jun 2019 14:34:49 +0000</resolved>
                                                    <fixVersion>Lustre 2.13.0</fixVersion>
                    <fixVersion>Lustre 2.12.3</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>5</watches>
                                                                            <comments>
                            <comment id="247889" author="adilger" created="Tue, 28 May 2019 20:47:10 +0000"  >&lt;p&gt;Update: there were no test failures in 2019-01 or 2018-12 so I thought that was the start of the failures, since it was failing about 5x per month after that.  However, searching further back there are again about 2-3 failures per month, and an old ticket &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10925&quot; title=&quot;sanity-flr test_33:  &amp;#39;&amp;#39;file content error: expected: ost1, actual: ost2&amp;#39;&amp;#39; &quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10925&quot;&gt;&lt;del&gt;LU-10925&lt;/del&gt;&lt;/a&gt; that shows the problem has existed for a long time already, going back to almost when the test was first landed.&lt;/p&gt;

&lt;p&gt;The first ~250 runs between 2017-08 and 2018-01 appear to be directly on the &lt;tt&gt;flr&lt;/tt&gt; branch under &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-9771&quot; title=&quot;FLR1: Landing tickets for File Level Redundancy Phase 1&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-9771&quot;&gt;&lt;del&gt;LU-9771&lt;/del&gt;&lt;/a&gt; and all pass.  The first failure is 2018-01 shortly after the FLR branch landed to master, with &lt;a href=&quot;https://review.whamcloud.com/30387&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch 20387&lt;/a&gt; &quot;&lt;tt&gt;&lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10287&quot; title=&quot;&amp;quot;lfs mirror verify&amp;quot; command&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10287&quot;&gt;&lt;del&gt;LU-10287&lt;/del&gt;&lt;/a&gt; flr: lfs mirror verify command&lt;/tt&gt;&quot; but looking at that patch it seems unlikely to be the culprit (the test does not use &quot;&lt;tt&gt;lfs mirror verify&lt;/tt&gt;&quot; at all, and that patch doesn&apos;t appear to affect any other code).&lt;/p&gt;

&lt;p&gt;In summary, it doesn&apos;t look like this can be isolated to a specific patch, and instead has to be isolated back from the test failure to see if it is a test bug or a code bug.&lt;/p&gt;</comment>
                            <comment id="247890" author="adilger" created="Tue, 28 May 2019 20:52:00 +0000"  >&lt;p&gt;It is a bit sad that we&apos;ve had this test failure for over 18 months and nobody who has hit the failure on their patch has bothered to file an LU ticket...&lt;/p&gt;</comment>
                            <comment id="247891" author="pfarrell" created="Tue, 28 May 2019 21:00:00 +0000"  >&lt;p&gt;The reason for this seems likely to be simple:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt; fail ost2 &amp;amp;
 sleep 1&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;It&apos;s clearly non-deterministic here.&lt;/p&gt;

&lt;p&gt;It seems to basically assume that ost2 will be unavailable for the subsequent operations, because it&apos;s being failed over in the background.&#160; Nothing is done to ensure that failover has either actually started (except that sleep) or that it has not completed yet.&lt;/p&gt;

&lt;p&gt;Seems simple enough - needs to be&#160;**&apos;stop&#160;ost2&apos; like it does &apos;stop ost1&apos; above.&lt;/p&gt;

&lt;p&gt;I&apos;ll push a patch.&lt;/p&gt;</comment>
                            <comment id="247894" author="gerrit" created="Tue, 28 May 2019 21:06:30 +0000"  >&lt;p&gt;Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/34985&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/34985&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-12350&quot; title=&quot;sanity-flr test_33: file content error: expected: ost1, actual: ost2&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-12350&quot;&gt;&lt;del&gt;LU-12350&lt;/del&gt;&lt;/a&gt; tests: Do not use background failover&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 2bd3fe05689017d26996a02d2231930dd67255ba&lt;/p&gt;</comment>
                            <comment id="248171" author="gerrit" created="Sat, 1 Jun 2019 03:55:47 +0000"  >&lt;p&gt;Oleg Drokin (green@whamcloud.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/34985/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/34985/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-12350&quot; title=&quot;sanity-flr test_33: file content error: expected: ost1, actual: ost2&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-12350&quot;&gt;&lt;del&gt;LU-12350&lt;/del&gt;&lt;/a&gt; tests: Do not use background failover&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 4ac0324fb9d824915b3dd11b75e81e609d9e8e84&lt;/p&gt;</comment>
                            <comment id="248197" author="pjones" created="Sat, 1 Jun 2019 14:34:49 +0000"  >&lt;p&gt;Landed for 2.13&lt;/p&gt;</comment>
                            <comment id="248571" author="gerrit" created="Thu, 6 Jun 2019 17:22:00 +0000"  >&lt;p&gt;Minh Diep (mdiep@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/35086&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/35086&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-12350&quot; title=&quot;sanity-flr test_33: file content error: expected: ost1, actual: ost2&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-12350&quot;&gt;&lt;del&gt;LU-12350&lt;/del&gt;&lt;/a&gt; tests: Do not use background failover&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_12&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: c7494b6ef36a1e831d3ccac07c66204538a8130c&lt;/p&gt;</comment>
                            <comment id="249536" author="gerrit" created="Thu, 20 Jun 2019 03:56:15 +0000"  >&lt;p&gt;Oleg Drokin (green@whamcloud.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/35086/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/35086/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-12350&quot; title=&quot;sanity-flr test_33: file content error: expected: ost1, actual: ost2&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-12350&quot;&gt;&lt;del&gt;LU-12350&lt;/del&gt;&lt;/a&gt; tests: Do not use background failover&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_12&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 020f774e0b0ff0f96173655744d976beb5af4a83&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="51870">LU-10925</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i00h47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>