https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4334 Lustre <p>When trying to declare two mgsnodes for HA only one entry is ever accepted when using backfstype=zfs. All below examples fail, if the --mgsnode=nid:nid syntax worked then all might be okay.</p> <p>mkfs.lustre --reformat --fsname=RSF1 --ost --index=2 --mgsnode=10.82.0.9@tcp1 --mgsnode=10.82.0.9@tcp1:10.82.0.10@tcp1 --servicenode=10.82.0.11@tcp1:10.82.0.12@tcp1 --backfstype=zfs OST2/ost</p> <p>mkfs.lustre --reformat --fsname=RSF1 --ost --index=2 --mgsnode=10.82.0.9@tcp1:10.82.0.10@tcp1 --servicenode=10.82.0.11@tcp1:10.82.0.12@tcp1 --backfstype=zfs OST2/ost</p> SL6.4 LU-4334

With ZFS can only declare a single mgsnode for MDT or OST

Bug Critical Resolved Fixed Nathaniel Clark Eric Kolb llnl prz zfs Mon, 2 Dec 2013 18:44:28 +0000 Mon, 3 Oct 2016 22:50:42 +0000 Thu, 11 Sep 2014 19:33:15 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.2 Lustre 2.7.0 Lustre 2.5.4 0 14 <p>Perhaps the below sequence displays the issue more clearly.</p> <p>$ tunefs.lustre OST2/ost<br/> checking for existing Lustre data: found</p> <p> Read previous values:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.9@tcp1</p> <p> Permanent disk data:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.9@tcp1</p> <p>Writing OST2/ost properties<br/> lustre:version=1<br/> lustre:flags=4098<br/> lustre:index=2<br/> lustre:fsname=RSF1<br/> lustre:svname=RSF1-OST0002<br/> lustre:failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1<br/> lustre:mgsnode=10.82.0.9@tcp1</p> <p>$ tunefs.lustre --mgsnode=10.82.0.9@tcp1 --mgsnode=10.82.0.10@tcp1 OST2/ost<br/> checking for existing Lustre data: found</p> <p> Read previous values:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.9@tcp1</p> <p> Permanent disk data:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.9@tcp1 mgsnode=10.82.0.10@tcp1</p> <p>Writing OST2/ost properties<br/> lustre:version=1<br/> lustre:flags=4098<br/> lustre:index=2<br/> lustre:fsname=RSF1<br/> lustre:svname=RSF1-OST0002<br/> lustre:failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1<br/> lustre:mgsnode=10.82.0.9@tcp1<br/> lustre:mgsnode=10.82.0.10@tcp1</p> <p>$ tunefs.lustre OST2/ost<br/> checking for existing Lustre data: found</p> <p> Read previous values:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.10@tcp1</p> <p> Permanent disk data:<br/> Target: RSF1-OST0002<br/> Index: 2<br/> Lustre FS: RSF1<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1 mgsnode=10.82.0.10@tcp1</p> <p>Writing OST2/ost properties<br/> lustre:version=1<br/> lustre:flags=4098<br/> lustre:index=2<br/> lustre:fsname=RSF1<br/> lustre:svname=RSF1-OST0002<br/> lustre:failover.node=10.82.0.11@tcp1:10.82.0.12@tcp1<br/> lustre:mgsnode=10.82.0.10@tcp1</p> <p>Hi, this syntax works. --mgsnode=node1:node2</p> <ol> <li>tunefs.lustre lustre1-ost4/ost0<br/> checking for existing Lustre data: found</li> </ol> <p> Read previous values:<br/> Target: lustre1-OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.3@o2ib failover.node=10.225.4.4@o2ib</p> <p> Permanent disk data:<br/> Target: lustre1:OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.3@o2ib failover.node=10.225.4.4@o2ib</p> <p>Writing lustre1-ost4/ost0 properties<br/> lustre:version=1 <br/> lustre:flags=4130 <br/> lustre:index=4<br/> lustre:fsname=lustre1<br/> lustre:svname=lustre1:OST0004<br/> lustre:mgsnode=10.225.8.3@o2ib<br/> lustre:failover.node=10.225.4.4@o2ib</p> <ol> <li>tunefs.lustre --mgsnode=mds1-225@o2ib:mds2-225@o2ib lustre1-ost4/ost0<br/> checking for existing Lustre data: found</li> </ol> <p> Read previous values:<br/> Target: lustre1-OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.3@o2ib failover.node=10.225.4.4@o2ib</p> <p> Permanent disk data:<br/> Target: lustre1:OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.3@o2ib failover.node=10.225.4.4@o2ib mgsnode=10.225.8.2@o2ib:10.225.8.3@o2ib</p> <p>Writing lustre1-ost4/ost0 properties<br/> lustre:version=1 <br/> lustre:flags=4130 <br/> lustre:index=4<br/> lustre:fsname=lustre1<br/> lustre:svname=lustre1:OST0004<br/> lustre:mgsnode=10.225.8.3@o2ib<br/> lustre:failover.node=10.225.4.4@o2ib<br/> lustre:mgsnode=10.225.8.2@o2ib:10.225.8.3@o2ib</p> <ol> <li>tunefs.lustre lustre1-ost4/ost0<br/> checking for existing Lustre data: found</li> </ol> <p> Read previous values:<br/> Target: lustre1-OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.2@o2ib:10.225.8.3@o2ib failover.node=10.225.4.4@o2ib</p> <p> Permanent disk data:<br/> Target: lustre1:OST0004<br/> Index: 4<br/> Lustre FS: lustre1 <br/> Mount type: zfs<br/> Flags: 0x1022 <br/> (OST first_time no_primnode )<br/> Persistent mount opts:<br/> Parameters: mgsnode=10.225.8.2@o2ib:10.225.8.3@o2ib failover.node=10.225.4.4@o2ib</p> <p>Writing lustre1-ost4/ost0 properties<br/> lustre:version=1 <br/> lustre:flags=4130 <br/> lustre:index=4<br/> lustre:fsname=lustre1<br/> lustre:svname=lustre1:OST0004<br/> lustre:mgsnode=10.225.8.2@o2ib:10.225.8.3@o2ib<br/> lustre:failover.node=10.225.4.4@o2ib</p> <p>Hello,</p> <p>Yes the --mgsnode=nid:nid setting can be applied to the MDTs and OSTs but the fail-over does not occur. The Lustre components seem only to use the fist nid in the specified list and upon fail-over of the MGS they will not use the second nid specified. </p> <p>It looks like there are two separate issues here:</p> <p>1) listing failover or mgsnode in the form --mgsnode=NID1 --mgsnode=NID2 or --mgsnode=NID1,NID2 will result in only NID2 being recorded.</p> <p>This seems to be due to how metadata is stored on zfs, that property names are unique, thus setting it twice will just overwrite the first with the second.</p> <p>2) listing nids in the form --mgsnode=NID1:NID2 will result in only NID1 being used</p> <p>As part of fixing this issue, we need to make certain that the relevant OSD documentation is updated to clearly define the APIs and expecations for the OSD developer.</p> <p>It seems like the right idea would be to store NID information in a single ZFS property &lt;server1ip1&gt;@tcp,&lt;server1ip2&gt;@tcp:&lt;server2ip1&gt;@tcp,&lt;server2ip2&gt;@tcp similar to how it can be input on the command line.</p> <p>This will apply to mgsnode, failnode, and servicenode.</p> <p>This will fix setting multiple mgsnode properties on ZFS.<br/> <a href="http://review.whamcloud.com/11161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11161</a></p> <p><b>Workaround for older systems:</b></p> <p>Instead of</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">tunefs.lustre --mgsnode=192.168.139.10@tcp --mgsnode=192.168.139.70@tcp mdt/mdt1 </pre> </div></div> <p>Use the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">zfs set lustre:mgsnode=192.168.139.10@tcp:192.168.139.70@tcp mdt/mdt1 </pre> </div></div> <p>Landed for 2.7</p> <p>Does</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">zfs set lustre:mgsnode=192.168.139.10@tcp:192.168.139.70@tcp mdt/mdt1 </pre> </div></div> <p>imply that ':' is the standard separator for zfs properties, for example in the <a href="https://jira.whamcloud.com/browse/LU-4749" title="ZFS-backed OST mkfs.lustre --servicenode does not correctly add failover_nids" class="issue-link" data-issue-key="LU-4749"><del>LU-4749</del></a> does failnode be split the same way.</p> Related LU-4749 LU-8311 Development Epic/Theme zfs Rank 1|hzwae7: Rank (Obsolete) 11857 Severity