https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12274 Lustre <p>I tried running tunefs.lustre and successfully changed the failover NIDs to what they should be. This problem is happening on several OSTs, but fixing one should fix them all. </p> <p>I'm assuming I forgot a step when I ran tunefs.lustre. </p> <p>tunefs.lustre --erase-param failover.node --param <span class="nobr"><a href="mailto:failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1" class="external-link" target="_blank" rel="nofollow noopener">failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> /dev/mapper/mpathg </p> <p>The OST OST0017 is mounted on 172.17.1.103 with the following parameters: </p> <p><span class="error">&#91;root@apslstr03 ~&#93;</span># tunefs.lustre --dryrun /dev/mapper/mpathg <br/> checking for existing Lustre data: found <br/> Reading CONFIGS/mountdata </p> <p>   Read previous values: <br/> Target:     lustrefc-OST0017 <br/> Index:      23 <br/> Lustre FS:  lustrefc <br/> Mount type: ldiskfs <br/> Flags:      0x2 <br/>               (OST ) <br/> Persistent mount opts: ,errors=remount-ro <br/> Parameters:  <span class="nobr"><a href="mailto:failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1" class="external-link" target="_blank" rel="nofollow noopener">failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="mailto:mgsnode=172.17.1.112@o2ib,172.16.1.112@tcp1" class="external-link" target="_blank" rel="nofollow noopener">mgsnode=172.17.1.112@o2ib,172.16.1.112@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="mailto:mgsnode=172.17.1.113@o2ib,172.16.1.113@tcp1" class="external-link" target="_blank" rel="nofollow noopener">mgsnode=172.17.1.113@o2ib,172.16.1.113@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>   Permanent disk data: <br/> Target:     lustrefc-OST0017 <br/> Index:      23 <br/> Lustre FS:  lustrefc <br/> Mount type: ldiskfs <br/> Flags:      0x2 <br/>               (OST ) <br/> Persistent mount opts: ,errors=remount-ro <br/> Parameters:  <span class="nobr"><a href="mailto:failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1" class="external-link" target="_blank" rel="nofollow noopener">failover.node=172.17.1.103@o2ib,172.16.1.103@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="mailto:mgsnode=172.17.1.112@o2ib,172.16.1.112@tcp1" class="external-link" target="_blank" rel="nofollow noopener">mgsnode=172.17.1.112@o2ib,172.16.1.112@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="mailto:mgsnode=172.17.1.113@o2ib,172.16.1.113@tcp1" class="external-link" target="_blank" rel="nofollow noopener">mgsnode=172.17.1.113@o2ib,172.16.1.113@tcp1^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>exiting before disk write. <br/> <span class="error">&#91;root@apslstr03 ~&#93;</span># </p> <p>However, the clients are still displaying errors like this: </p> <p>May  8 11:43:33 localhost kernel: Lustre: 2028:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: <span class="error">&#91;sent 1557333772/real 0&#93;</span> req@ffff880bd9296f00 x1632920191594624/t0(0) o8-&gt;<span class="nobr"><a href="mailto:lustrefc-OST0017-osc-ffff8817ef372000@172.17.1.106@o2ib:28/4" class="external-link" target="_blank" rel="nofollow noopener">lustrefc-OST0017-osc-ffff8817ef372000@172.17.1.106@o2ib:28/4^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> lens 520/544 e 0 to 1 dl 1557333813 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 <br/> May  8 11:43:33 localhost kernel: Lustre: 2028:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 65 previous similar messages <br/> May  8 11:45:26 localhost kernel: LNet: 1994:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 172.17.1.106@o2ib: 3 seconds <br/> May  8 11:45:26 localhost kernel: LNet: 1994:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 39 previous similar messages</p> Servers: Lustre-2.10.2, Kernel: 3.10.0-693.5.2.el7_lustre.x86_64 <br/> Clients: Lustre-2.10.3, Kernel: 3.10.0-693.21.1.el7.x86_64 <br/> Client/Server OS: CentOS Linux release 7.4.1708 LU-12274

Clients aren't connecting to OST defined failover.node

Question/Request Blocker Open Unresolved Sebastien Buisson Roger Sersted Wed, 8 May 2019 19:47:12 +0000 Sun, 12 May 2019 14:38:37 +0000 Lustre 2.10.2 0 4 <p>Sebastien</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>Hi,</p> <p>Did you run the tunes.lustre commands while the targets were stopped (ie unmounted)?</p> <p>I unmounted the OSTs inquestion.  OSTs not being modified were mounted.  The MDT and MGT were both mounted.</p> <p>I think we will need to have a look at the Lustre Logs on the MGS.<br/> Could you please run the following commands on the MGS and attach the lustrefc-client.txt file to this ticket (output of llog_reader)?</p> <p>Assuming your Lustre file system name is lustrefc, that would be:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mgs# debugfs -c -R 'dump CONFIGS/lustrefc-client /tmp/lustrefc-client' &lt;mgt device&gt; mgs# llog_reader /tmp/lustrefc-client &gt; /tmp/lustrefc-client.txt </pre> </div></div> <p>Thanks.</p> <p>I have attached the requested output. I should add, my cluster is down due to this problem.</p> <p>As the name suggests, failover node parameter serves the purpose of specifying failover NIDs for targets. It does not reflect the primary NID of a target.</p> <p>So now that you mention that your cluster is down, I am wondering if your targets have been moved so that their primary NID is now different. If the targets did not move and they all run on their primary node, then this problem with the failover node change should not lead to any downtime.</p> <p>I'm in the process of updating the Lustre servers.  I unmounted the OSTs on one set of servers and mounted them on to their HA partners.  I have done this in the past with one node and it worked fine.  I have 6 OSSes configured in HA pairs.  I am not running any HA software.  If a server fails, I manually unmount and then mount to the HA partner. </p> <p>I can see in the llog_reader output that target lustrefc-OST0017 for instance is still registered with NIDs 172.17.1.105 and 172.17.1.106. It explains the error messages on the clients.</p> <p>I just noticed this important message in the Lustre Operations Manual:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>If a --failnode option is added to a target to designate a failover server for the target, the target must be re-mounted on the primary node before the --failnode option takes effect </pre> </div></div> <p>So the problem you are facing could be due to the fact that after tunefs.lustre, the target was directly mounted on the failover node. For instance on 172.17.1.103 for target lustrefc-OST0017. Does it make sense?</p> <p>Targets should be mounted on the primary node right after a tunefs.lustre that changes the failnodes, and only after that failed over to a secondary node.</p> <p>Great catch on that.  I went through all of my OSTs and remounted and checked the failover setting.  I then remounted to the HA partner and the filesystem is working.  Quick question, "How would I change primary NID of an OST?"  Would I specify the "servicenode" option?</p> <p>Good to hear!</p> <p>If you need to change a primary nid, I would advise to follow the dedicated instructions in the Lustre Operations Manual:<br/> <a href="http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.changingservernid" class="external-link" target="_blank" rel="nofollow noopener">http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.changingservernid</a></p> Development Epic/Theme Lustre-2.10.2 Rank 1|i00fzj: Rank (Obsolete) 9223372036854775807