https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-885 Lustre <p>Running recovery-mds-scale FLAVOR=mds for about 2 hours(MDS fail over 14 times), network is not available for standby MDS server and it cannot be access after that even doing power cycle. I got this similar issue twice.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 14 times, and counting... sleeping 501 seconds ... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=7904 DURATION=86400 PERIOD=600 Wait mds1 recovery complete before doing next failover .... affected facets: mds1 client-6: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover client-18: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec client-12: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec client-18: cannot run remote command on client-12,client-13,client-17,client-18 with client-12: cannot run remote command on client-12,client-13,client-17,client-18 with client-17: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec client-17: cannot run remote command on client-12,client-13,client-17,client-18 with client-13: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec client-13: cannot run remote command on client-12,client-13,client-17,client-18 with Starting failover on mds1 Failing mds1 on node client-6 + pm -h powerman --off client-6 Command completed successfully affected facets: mds1 + pm -h powerman --on client-6 Command completed successfully Failover mds1 to client-2 15:35:30 (1322609730) waiting for client-2 network 900 secs ... waiting ping -c 1 -w 3 client-2, 895 secs left ... waiting ping -c 1 -w 3 client-2, 890 secs left ... waiting ping -c 1 -w 3 client-2, 885 secs left ... waiting ping -c 1 -w 3 client-2, 880 secs left ... waiting ping -c 1 -w 3 client-2, 875 secs left ... waiting ping -c 1 -w 3 client-2, 870 secs left ... waiting ping -c 1 -w 3 client-2, 865 secs left ... waiting ping -c 1 -w 3 client-2, 860 secs left ... waiting ping -c 1 -w 3 client-2, 855 secs left ... waiting ping -c 1 -w 3 client-2, 850 secs left ... waiting ping -c 1 -w 3 client-2, 845 secs left ... waiting ping -c 1 -w 3 client-2, 840 secs left ... waiting ping -c 1 -w 3 client-2, 835 secs left ... waiting ping -c 1 -w 3 client-2, 830 secs left ... waiting ping -c 1 -w 3 client-2, 825 secs left ... waiting ping -c 1 -w 3 client-2, 820 secs left ... </pre> </div></div> lustre-master build #353 RHEL6-x8_64 for both server and client LU-885

recovery-mds-scale (FLAVOR=mds) fail, network is not avaliable

Bug Minor Resolved Cannot Reproduce WC Triage Sarah Liu Tue, 29 Nov 2011 19:01:26 +0000 Thu, 3 Oct 2019 17:41:18 +0000 Mon, 29 May 2017 02:52:04 +0000 Lustre 2.3.0 0 4 <p>recovery-radom-scale failed after running about 4 hours(mds1 failed over 24 times) hit the similar issue. the standby mds server is not usable.</p> <p>waiting ping -c 1 -w 3 client-7, 5 secs left ...<br/> Network not available!<br/> 2011-12-01 01:39:58 Terminating clients loads ...<br/> Duration: 86400<br/> Server failover period: 600 seconds<br/> Exited after: 14119 seconds<br/> Number of failovers before exit:<br/> mds1 failed over 24 times<br/> Status: FAIL: rc=1</p> <p>So it appears that the failover node is not coming up for some reason (pings not working and such), somebody need to reproduce this and then check into what's going on at the failover node? vm failed to start? Some rootfs corruption so that it's stuck during booting waiting for root password or whatever?</p> <p>Potentially a TT ticket if the node fails to start.</p> <p>Actually in both recovery-mds-scale(FLAVOR=mds) and recovery-radom-scale tests, the failover node was not accessible after rebooting for several times. Robert was in the lab and helped on this. As he seen, the nodes need a physical power on and it seems nothing special.</p> <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>this seems to be a problem related to the tool pm, for the nodes were not stuck and only need a power on</p> <p>Hongchao how did you replicate this, and have we seen this recently under autotest? If so can you post links to the results here.</p> <p>Adding Hongchao as a watcher so he sees Chris's question</p> <p>Hi Chris,<br/> I can't replicate this issue, and there is no new occurrence under autotest recently.</p> <p>As per Sarah has not reocurred for last three tags so removing as a blocker</p> <p>Close old ticket.</p> Related LU-893 Development Rank 1|hzw0tz: Rank (Obsolete) 10215 Severity