https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7428 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/5d42a610-8187-11e5-a41e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5d42a610-8187-11e5-a41e-5254006e85c2</a>.</p> <p>The sub-test test_84 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Update not seen after 90s: wanted '' got 'lustre:MDT0000' conf-sanity test_84: @@@@@@ FAIL: /dev/lvm-Role_MDS/P1 failed to initialize! Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4843:error() = /usr/lib64/lustre/tests/test-framework.sh:1270:mount_facet() = /usr/lib64/lustre/tests/test-framework.sh:1188:mount_facets() = /usr/lib64/lustre/tests/test-framework.sh:2513:facet_failover() = /usr/lib64/lustre/tests/conf-sanity.sh:5594:test_84() = /usr/lib64/lustre/tests/test-framework.sh:5090:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5127:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4992:run_test() </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: conf-sanity 84<br/> Info required for matching: replay-dual 0a</p> LU-7428

conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!

Bug Major Resolved Fixed James A Simmons Maloo p4hc Sun, 15 Nov 2015 05:28:17 +0000 Mon, 29 May 2017 05:58:11 +0000 Mon, 26 Sep 2016 15:44:03 +0000 Lustre 2.8.0 Lustre 2.9.0 0 18 <p>This is failing about twice per day on master.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17371" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17371</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> tests: write superblock in conf-sanity test_84<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d3e516e59697f7c48e5cb97054ee04cf97dc7132</p> <p>It is another failure instance <a href="https://jira.whamcloud.com/browse/LU-7169" title="conf-sanity 84 restart mds1 failed" class="issue-link" data-issue-key="LU-7169"><del>LU-7169</del></a>.</p> <p>Please don't close this bug, as I've got a patch tracked here that may fix the problem.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17371/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17371/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> tests: write superblock in conf-sanity test_84<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5fda01f3002e7e742a206ce149652c6b78356828</p> <p>It looks like conf-sanity test_84 is still failing in some cases, even with this patch applied:</p> <p>It still fails consistently with the new e2fsprogs patches base on master build d059b3c01 "<a href="https://jira.whamcloud.com/browse/LU-6693" title="object_update_param_get inconsistent return value" class="issue-link" data-issue-key="LU-6693"><del>LU-6693</del></a> out: not return NULL in object_update_param_get", though I can't really see why except that the superblock label never gets updated:<br/> <a href="http://review.whamcloud.com/17151" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17151</a><br/> <a href="http://review.whamcloud.com/17150" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17150</a><br/> <a href="http://review.whamcloud.com/17152" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17152</a><br/> <a href="http://review.whamcloud.com/17153" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17153</a></p> <p>Bob, this test is failing too often (see many different linked bugs), and doesn't provide much value for testing in comparison. Could you please add it to the ALWAYS_EXCEPT list until it can be fixed. My last attempt didn't seem to resolve the problem.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17482" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17482</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: disable conf-sanity, test_84<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a26001d568432d1fa646c6fc850e0ab66e41f97f</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/17482/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17482/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: disable conf-sanity, test_84<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 74d95a078c6725884f67a2737ea5b7e55fab1087</p> <p>It looks like all of the e2fsprogs test failures are on CentOS 7. I don't know if that means the failures are <em>only</em> on CentOS 7, or if it is because the new e2fsprogs patches are not tested on any other distro.</p> <p>Status update:<br/> I have tried to reproduce the issue on CentOS7 with the latest e2fsprogs (branch remotes/origin/master-lustre), and I can't reproduce it.</p> <p>master, build# 3264, 2.7.64 tag<br/> recovery-mds-scale test_failover_ost failed with same issue.<br/> Hard Failover: EL7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/cf43bf1c-9e9a-11e5-b163-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cf43bf1c-9e9a-11e5-b163-5254006e85c2</a></p> <p>Hard Failover: EL7 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/a39034e8-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a39034e8-a077-11e5-8d69-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Regression:EL7.1 Server/EL7.1 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/7704baac-9f37-11e5-ba94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7704baac-9f37-11e5-ba94-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Regression:EL7.1 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/18ce6c0a-9f2b-11e5-bf9b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/18ce6c0a-9f2b-11e5-bf9b-5254006e85c2</a></p> <p>Hongchao</p> <p>Have the tests ever run successfully on autotest? Perhaps there is something missing from the TEI environment for running these tests that you can identify so the TEI team can correct?</p> <p>Peter</p> <p>Yes, there are successful test on autotest,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&amp;page=3&amp;sub_test%5Bquery_bugs%5D=&amp;sub_test%5Bstatus%5D=&amp;sub_test%5Bsub_test_script_id%5D=5e30cae0-7f56-11e4-b7e8-5254006e85c2&amp;test_node%5Barchitecture_type_id%5D=&amp;test_node%5Bdistribution_type_id%5D=0dcf0e82-e30f-11e4-9cb2-5254006e85c2&amp;test_node%5Bfile_system_type_id%5D=&amp;test_node%5Blustre_branch_id%5D=&amp;test_node%5Bos_type_id%5D=&amp;test_node_network%5Bnetwork_type_id%5D=&amp;test_session%5Bend_date%5D=&amp;test_session%5Bquery_recent_period%5D=&amp;test_session%5Bstart_date%5D=&amp;test_session%5Btest_group%5D=&amp;test_session%5Btest_host%5D=&amp;test_session%5Buser_id%5D=&amp;test_set%5Btest_set_script_id%5D=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;utf8=✓&amp;warn%5Bnotice%5D=true </pre> </div></div> <p>I'm not sure whether the problem is related to TEI or not.</p> <p>Another instance for EL7.1 Server/EL7.1 Client - DNE<br/> Master , Build# 3270<br/> <a href="https://testing.hpdd.intel.com/test_sets/6cae7cf8-a26d-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6cae7cf8-a26d-11e5-bdef-5254006e85c2</a></p> <p>Hongchao, it looks like the main problem is that the e2label command run as part of mkfs.lustre to update the filesystem label from <tt>fsname:MDT0000</tt> to <tt>fsname-MDT0000</tt> is failing to be written to disk. It isn't clear if that is a problem with the way the dev_rdonly code works, that prevents the superblock update from being persistent, or if there is something in the VM that is dropping the writes from the guest.</p> <p>What would make sense is to add some debugging to conf-sanity.sh test_84 (since this one seems to hit this failure most often) that runs <tt>e2label $MDSDEV</tt> to print the filesystem label after the initial mount + sync + sleep but before <tt>replay_barrier</tt>, and then print it again in <tt>mount_facet</tt> if the mount has failed so that we can see what the current label is.</p> <ul class="alternate" type="square"> <li>We do see similar failure during relay-dual tests on <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Configuration : 4 node cluster . 1 MDS/ 1 OSS / 2 Clients. Release 3.10.0_229.20.1.el7_lustremaster_master__81.x86_64_g70bb27b Server 2.7.64 Client 2.7.642.7.64 kernel - 3.10.0_229.20.1.el7 git hash - 70bb27b </pre> </div></div></li> </ul> <ul class="alternate" type="square"> <li>stdout.log <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-dual test 0a: expired recovery with lost client ============================================ 05:44:27 (1450935867) Check file is LU482_FAILED=/tmp/replay-dual.lu482.wNwVYw Filesystem 1K-blocks Used Available Use% Mounted on 192.168.113.21@tcp:/lustre 1345184 35144 1209424 3% /mnt/lustre total: 50 creates in 0.14 seconds: 355.12 creates/second fail_loc=0x80000514 Failing mds1 on fre1321 Stopping /mnt/mds1 (opts:) on fre1321 pdsh@fre1323: fre1321: ssh exited with exit code 1 reboot facets: mds1 Failover mds1 to fre1321 05:44:45 (1450935885) waiting for fre1321 network 900 secs ... 05:44:45 (1450935885) network interface is UP mount facets: mds1 Starting mds1: -o rw,user_xattr /dev/vdb /mnt/mds1 Waiting 90 secs for update Waiting 80 secs for update Waiting 70 secs for update Waiting 60 secs for update Waiting 50 secs for update Waiting 40 secs for update Waiting 30 secs for update Waiting 20 secs for update Waiting 10 secs for update Update not seen after 90s: wanted '' got 'lustre:MDT0000' replay-dual test_0a: @@@@@@ FAIL: /dev/vdb failed to initialize! Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4822:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4853:error() = /usr/lib64/lustre/tests/test-framework.sh:1270:mount_facet() = /usr/lib64/lustre/tests/test-framework.sh:1188:mount_facets() = /usr/lib64/lustre/tests/test-framework.sh:2523:facet_failover() = /usr/lib64/lustre/tests/replay-dual.sh:66:test_0a() = /usr/lib64/lustre/tests/test-framework.sh:5100:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5137:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4954:run_test() = /usr/lib64/lustre/tests/replay-dual.sh:76:main() Dumping lctl log to /tmp/test_logs/1450935862/replay-dual.test_0a.*.1450935987.log fre1324: Warning: Permanently added 'fre1323,192.168.113.23' (ECDSA) to the list of known hosts. fre1322: Warning: Permanently added 'fre1323,192.168.113.23' (ECDSA) to the list of known hosts. fre1321: Warning: Permanently added 'fre1323,192.168.113.23' (ECDSA) to the list of known hosts. FAIL 0a (121s) </pre> </div></div></li> <li>Its reproducible. please note the kernel version is 3.10.x</li> <li>Attaching the logs, let me know, if more information is required.</li> </ul> <p>Can somebody confirm, if its the same issue ?</p> <p>Thanks</p> <p>Another instance found for hardfailover: EL7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/285d11ca-bc00-11e5-a592-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/285d11ca-bc00-11e5-a592-5254006e85c2</a></p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18178" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18178</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4474b31c11f7b9034a25371abbaafe61371e9b85</p> <p>Encountered same issue for tag 2.7.66 for FULL- EL7.1 Server/EL6.7 Client , master , build# 3314 for replay-vbr.<br/> <a href="https://testing.hpdd.intel.com/test_sets/85c15246-ca91-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/85c15246-ca91-11e5-9609-5254006e85c2</a><br/> Encountered another instance for tag 2.7.66 for FUL - EL7.1 Server/EL7.1 Client , master , build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/b9fde76c-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9fde76c-ca88-11e5-84d3-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Update not seen after 90s: wanted '' got 'lustre:MDT0000' replay-vbr test_1b: @@@@@@ FAIL: /dev/lvm-Role_MDS/P1 failed to initialize! </pre> </div></div> <p>Another failure for master : Tag 2.7.66 FULL - EL7.1 Server/SLES11 SP3 Client, build# 3314 for replay-single.<br/> <a href="https://testing.hpdd.intel.com/test_sets/9fad2e16-ca7b-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9fad2e16-ca7b-11e5-9609-5254006e85c2</a></p> <p>Another instance for FULL - EL7.1 Server/EL7.1 Client - DNE, master, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/a973fa52-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a973fa52-cac5-11e5-9609-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9ece523c-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ece523c-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/Client, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/SLES11 SP3 Client, tag 2.7.66, master build 3316<br/> <a href="https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL7.1 Server/EL6.7 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/85c15246-ca91-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/85c15246-ca91-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL7.1 Server/EL7.1 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b9fde76c-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9fde76c-ca88-11e5-84d3-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/SLES11 SP3 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/a63f3418-ca7b-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a63f3418-ca7b-11e5-9609-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9fad2e16-ca7b-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9fad2e16-ca7b-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/a973fa52-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a973fa52-cac5-11e5-9609-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9ece523c-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ece523c-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found on b2_8 for failover testing , build# 6.<br/> <a href="https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2</a></p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18871" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18871</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: commit the label change to disk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9c6eb1c7f48e205168bb9f3a011af8c957b19616</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18871/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18871/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: commit the label change to disk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4635d6235f8c8f4bb212aa59710c4f68db6acd7a</p> <p>Additional fix has landed to master for 2.9.0</p> <p>The patch landed to try the new workaround, but test 84 is still in the ALWAYS_EXCEPT list in conf-sanity.sh, so until that is removed there is no way to know whether this problem was actually fixed.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20194" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20194</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: remove test 84 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 589c22a31431c94fcd981c95847d295f841fb1e8</p> <p>It might be worthwhile to test the patch from <a href="https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b</a> to see if that is fixing the problem? I think that patch could be simplified to just call <tt>sb-&gt;s_op-&gt;s_freeze()</tt> before marking the device read-only.</p> <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/20535" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20535</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> osd: freeze fs before set device readonly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cb2a762f99cef9ff86dc76445248f91b40f0199b</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20586" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20586</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> osd: set rdonly correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 521b8290fbf0b47d4ad03272a206d038f648db2d</p> <p>the patch ported from MRP-2135 (<a href="https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b</a>) <br/> is tracked at <a href="http://review.whamcloud.com/20586" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20586</a></p> <p>Does this patch mean we don't need <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> anymore?</p> <p>No, this won't replace <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a>.</p> <p>This patch is to (hopefully) fix a problem where the device is sync'd and set read-only, but loses some recent writes, for an unknown reason. This shows up with a variety of different symptoms, and may be a result of bad interactions with LVM and VM virtual block devices, or it may be caused by the dev readonly patches.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20586/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20586/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> osd: set device read-only correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a079ade7913b923b795ea5c01df4e69bf1a87691</p> <p>Landed for 2.9</p> <p>Did the landing of <a href="http://review.whamcloud.com/20586/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20586/</a> resolve this issue? I see that we still have 84 in <tt>ALWAYS_EXCEPT</tt>.</p> <p>the patch <a href="http://review.whamcloud.com/#/c/20194/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/20194/</a> to remove test 84 has been refreshed, and have passed the tests in Maloo now.</p> <p>Reopen until patch enabling test_84 actually lands.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20194/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20194/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> test: remove test 84 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e40a3fd8a531ed60528ca82e02ce41918b1be6ba</p> <p>Test re-enabled for 2.9</p> <p>Hi Hongchao,</p> <p>With patch <a href="http://review.whamcloud.com/7200" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7200</a> on master branch, conf-sanity test 84 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-31vm7 e2label /dev/mapper/mds1_flakey 2&gt;/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Update not seen after 90s: wanted '' got 'lustre:MDT0000' conf-sanity test_84: @@@@@@ FAIL: /dev/mapper/mds1_flakey failed to initialize! </pre> </div></div> <p><a href="https://testing.hpdd.intel.com/test_sets/e88a61c2-89bf-11e6-a8b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e88a61c2-89bf-11e6-a8b7-5254006e85c2</a></p> <p>Could you please advise? Thank you.</p> <p>Hi Hongchao,<br/> Since the above failure is specific to dm-flakey patch, I filed <a href="https://jira.whamcloud.com/browse/LU-8729" title="conf-sanity test_84: FAIL: /dev/mapper/mds1_flakey failed to initialize!" class="issue-link" data-issue-key="LU-8729"><del>LU-8729</del></a> to track it.</p> Duplicate LU-7481 Related LU-7364 LU-6029 LU-7169 LU-7097 LU-6992 LU-7222 LU-7368 LU-6789 LU-7361 LU-7492 LU-7100 LU-7509 Development Rank 1|hzxt2f: Rank (Obsolete) 9223372036854775807 Severity