https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8544 Lustre <p>This issue was created by maloo for Saurabh Tandan &lt;saurabh.tandan@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/72a0fd32-6033-11e6-aa74-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/72a0fd32-6033-11e6-aa74-5254006e85c2</a>.</p> <p>The sub-test test_pairwise_fail failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>start client on trevis-54vm5 failed </pre> </div></div> <p>test logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-54vm5 test -f /tmp/client-load.pid &amp;&amp; { kill -s TERM \$(cat /tmp/client-load.pid); rm -f /tmp/client-load.pid; } + pm -h powerman --off trevis-54vm5 Command completed successfully + pm -h powerman --on trevis-54vm5 Command completed successfully 14:43:16 (1470926596) waiting for trevis-54vm5 network 900 secs ... waiting ping -c 1 -w 3 trevis-54vm5, 895 secs left ... waiting ping -c 1 -w 3 trevis-54vm5, 890 secs left ... waiting ping -c 1 -w 3 trevis-54vm5, 885 secs left ... waiting ping -c 1 -w 3 trevis-54vm5, 880 secs left ... 14:43:48 (1470926628) network interface is UP CMD: trevis-54vm5 hostname pdsh@trevis-54vm1: trevis-54vm5: mcmd: connect failed: Connection refused CMD: trevis-54vm5 hostname Reintegrating trevis-54vm5 Starting client: trevis-54vm5: -o user_xattr,flock trevis-54vm7:trevis-54vm3:/lustre /mnt/lustre CMD: trevis-54vm5 mkdir -p /mnt/lustre CMD: trevis-54vm5 mount -t lustre -o user_xattr,flock trevis-54vm7:trevis-54vm3:/lustre /mnt/lustre CMD: trevis-54vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck\" \"all -lnet -lnd -pinger\" 4 trevis-54vm5: stat: cannot read file system information for ‘/mnt/lustre’: Input/output error recovery-double-scale test_pairwise_fail: @@@@@@ FAIL: start client on trevis-54vm5 failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4804:error() = /usr/lib64/lustre/tests/recovery-double-scale.sh:72:reboot_recover_node() = /usr/lib64/lustre/tests/recovery-double-scale.sh:160:failover_pair() = /usr/lib64/lustre/tests/recovery-double-scale.sh:251:test_pairwise_fail() = /usr/lib64/lustre/tests/test-framework.sh:5068:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5107:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4954:run_test() = /usr/lib64/lustre/tests/recovery-double-scale.sh:303:main() </pre> </div></div> LU-8544

recovery-double-scale test_pairwise_fail: start client on trevis-54vm5 failed

Bug Critical Resolved Fixed Hongchao Zhang Maloo Thu, 25 Aug 2016 22:37:20 +0000 Wed, 5 Aug 2020 21:35:19 +0000 Thu, 29 Sep 2016 17:40:57 +0000 Lustre 2.9.0 0 9 <p>This issue has been seen around 40 times in past 30 days overall.</p> <p>This issue was first seen for master on 2016-07-08 for build# 3405, Tag 2.8.55, Lustre version: 2.8.55.27.geb2657a<br/> <a href="https://testing.hpdd.intel.com/test_sets/e5a03c8e-4568-11e6-80b9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e5a03c8e-4568-11e6-80b9-5254006e85c2</a></p> <p>Hongchao</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Looking at test failures in Maloo, I see that this test started failing with this error message on 2016-07-01. I see failures on both onyx and trevis.</p> <p>the problem could be related to <a href="https://jira.whamcloud.com/browse/LU-7759" title="umount hanging in modern distros when OST is unavailable" class="issue-link" data-issue-key="LU-7759"><del>LU-7759</del></a>, and the default LL_SBI_LAZYSTATFS cause the "df" fail to wait the recovery to finish.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000004:0.0:1472256895.043480:0:3882:0:(obd_class.h:1166:obd_statfs_async()) lustre-clilov-ffff880037e6c000: osfs ffff88007aa25210 age 4294647497, max_age 4294916319 00020000:00080000:0.0:1472256895.043483:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 0 inactive 00020000:00080000:0.0:1472256895.043484:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 1 inactive 00020000:00080000:0.0:1472256895.043485:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 2 inactive 00020000:00080000:0.0:1472256895.043485:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 3 inactive 00020000:00080000:0.0:1472256895.043485:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 4 inactive 00020000:00080000:0.0:1472256895.043486:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 5 inactive 00020000:00080000:0.0:1472256895.043486:0:3882:0:(lov_request.c:648:lov_prep_statfs_set()) lov idx 6 inactive 00000080:00020000:0.0:1472256895.043488:0:3882:0:(llite_lib.c:1890:ll_statfs_internal()) obd_statfs fails: rc = -5 </pre> </div></div> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22459" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22459</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8544" title="recovery-double-scale test_pairwise_fail: start client on trevis-54vm5 failed" class="issue-link" data-issue-key="LU-8544"><del>LU-8544</del></a> test: using lfs df in client_up<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7438b526bb79b5138fb51cd1ed58eadc1bbeab26</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22459/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22459/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8544" title="recovery-double-scale test_pairwise_fail: start client on trevis-54vm5 failed" class="issue-link" data-issue-key="LU-8544"><del>LU-8544</del></a> test: using lfs df in client_up<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 72ec6eb3c74c85f54277aadfd9b83167ea8e81ec</p> <p>Landed for 2.9</p> <p>Hi Bruno,<br/> I will check it and update its status once I found something.</p> Related LU-5788 LU-6655 LU-7759 LU-6670 LU-4039 LU-8526 LU-8731 LU-6610 Development Rank 1|hzym3b: Rank (Obsolete) 9223372036854775807 Severity