https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8311 Lustre <p>We are unable to mount the targets on Lustre servers when using multirail configuration on the MGS.</p> <p><a href="https://jira.whamcloud.com/browse/LU-4334" title="With ZFS can only declare a single mgsnode for MDT or OST" class="issue-link" data-issue-key="LU-4334"><del>LU-4334</del></a> introduced a format change of the mgsnode value on the targets.</p> <p>Old format:<br/> <tt>mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1 mgsnode=192.168.101.42@tcp,192.168.102.42@tcp1</tt></p> <p>New format: <br/> <tt>mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1:192.168.101.42@tcp,192.168.102.42@tcp1</tt></p> <p>With patch <a href="https://jira.whamcloud.com/browse/LU-5690" title="Unable to parse the mdt.nosquash_nids parameter when using commas in expr_list" class="issue-link" data-issue-key="LU-5690"><del>LU-5690</del></a>, we are now unable to start any target with this new format. We can see this Lustre error in the console of the OSS: </p> <p><tt>LDISKFS-fs (vdb): Unrecognized mount option "192.168.102.41@tcp1:192.168.101.42@tcp" or missing value</tt></p> <p>The debug log reports the following message while trying to mount OST 0:</p> <blockquote> <p>00000020:01200004:0.0F:1466084531.621867:0:2966:0:(obd_mount.c:1339:lustre_fill_super()) VFS Op: sb ffff88001f583c00<br/> 00000020:01000004:0.0:1466084531.621882:0:2966:0:(obd_mount.c:830:lmd_print()) mount data:<br/> 00000020:01000004:0.0:1466084531.621883:0:2966:0:(obd_mount.c:833:lmd_print()) device: /dev/vdb<br/> 00000020:01000004:0.0:1466084531.621884:0:2966:0:(obd_mount.c:834:lmd_print()) flags: 0<br/> 00000020:01000004:0.0:1466084531.621884:0:2966:0:(obd_mount.c:837:lmd_print()) <b>options: errors=remount-ro,192.168.102.41@tcp1:192.168.101.42@tcp,192.168.102.42@tcp1</b><br/> 00000020:01000004:0.0:1466084531.621885:0:2966:0:(obd_mount.c:1386:lustre_fill_super()) Mounting server from /dev/vdb<br/> 00000020:01000004:0.0:1466084531.621887:0:2966:0:(obd_mount_server.c:1627:osd_start()) Attempting to start scratch-OST0000, type=osd-ldiskfs, lsifl=200002, mountfl=0<br/> 00000020:01000004:0.0:1466084531.621925:0:2966:0:(obd_mount.c:191:lustre_start_simple()) Starting obd scratch-OST0000-osd (typ=osd-ldiskfs)<br/> 00000004:00020000:0.0:1466084531.623545:0:2966:0:(osd_handler.c:5613:osd_mount()) scratch-OST0000-osd: can't mount /dev/vdb: -22<br/> 00000020:00020000:0.0:1466084531.624487:0:2966:0:(obd_config.c:572:class_setup()) setup scratch-OST0000-osd failed (-22)<br/> 00000020:00020000:0.0:1466084531.625290:0:2966:0:(obd_mount.c:200:lustre_start_simple()) scratch-OST0000-osd setup error -22<br/> 00000020:01000000:0.0:1466084531.626153:0:2966:0:(obd_config.c:750:class_decref()) finishing cleanup of obd scratch-OST0000-osd (scratch-OST0000-osd_UUID)<br/> 00000020:00020000:0.0:1466084531.626156:0:2966:0:(obd_mount_server.c:1701:server_fill_super()) Unable to start osd on /dev/vdb: -22<br/> 00000020:01000004:0.0:1466084531.627005:0:2966:0:(obd_mount.c:653:lustre_put_lsi()) put ffff88001f583c00 1<br/> 00000020:01000004:0.0:1466084531.627007:0:2966:0:(obd_mount.c:603:lustre_free_lsi()) Freeing lsi ffff880017c67000<br/> 00000020:00020000:0.0:1466084531.627009:0:2966:0:(obd_mount.c:1405:lustre_fill_super()) Unable to mount (-22)</p></blockquote> <p>This is easily reproducible with Lustre 2.5.3.90+<a href="https://jira.whamcloud.com/browse/LU-5690" title="Unable to parse the mdt.nosquash_nids parameter when using commas in expr_list" class="issue-link" data-issue-key="LU-5690"><del>LU-5690</del></a>.</p> <blockquote> <ol> <li>tunefs.lustre --erase-params --mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1 --mgsnode=192.168.101.42@tcp,192.168.102.42@tcp1 /dev/vdb<br/> checking for existing Lustre data: found<br/> Reading CONFIGS/mountdata</li> </ol> <p> Read previous values:<br/> Target: scratch-OST0000<br/> Index: 0<br/> Lustre FS: scratch<br/> Mount type: ldiskfs<br/> Flags: 0x42<br/> (OST update )<br/> Persistent mount opts: errors=remount-ro<br/> Parameters: mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1 mgsnode=192.168.101.42@tcp,192.168.102.42@tcp1</p> <p> Permanent disk data:<br/> Target: scratch-OST0000<br/> Index: 0<br/> Lustre FS: scratch<br/> Mount type: ldiskfs<br/> Flags: 0x42<br/> (OST update )<br/> Persistent mount opts: errors=remount-ro<br/> Parameters: mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1:192.168.101.42@tcp,192.168.102.42@tcp1</p> <p>Writing CONFIGS/mountdata</p> <ol> <li>mount -t lustre /dev/vdb /mnt/fs/scratch/ost0<br/> mount.lustre: set /sys/block/vdb/queue/max_sectors_kb to 2147483647</li> </ol> <p>mount.lustre: mount /dev/vdb at /mnt/fs/scratch/ost0 failed: Invalid argument<br/> This may have multiple causes.<br/> Are the mount options correct?<br/> Check the syslog for more info.</p></blockquote> Lustre 2.5.3.90 w/ Bull patches, including <a href="https://jira.whamcloud.com/browse/LU-5690" title="Unable to parse the mdt.nosquash_nids parameter when using commas in expr_list" class="issue-link" data-issue-key="LU-5690"><strike>LU-5690</strike></a> LU-8311

Target does not mount with the new mgsnode parameter format in case of multirail configuration

Bug Major Resolved Fixed Jian Yu Bruno Travouillon p4b Tue, 21 Jun 2016 13:40:46 +0000 Thu, 14 Jun 2018 21:41:17 +0000 Sat, 29 Oct 2016 00:24:15 +0000 Lustre 2.5.3 Lustre 2.9.0 0 10 <p>Hi Jian,</p> <p>Can you please look into this issue?</p> <p>Thanks.<br/> Joe</p> <p>For the record, an easy workaround is:</p> <p>&#35; tunefs.lustre --erase-params --param mgsnode=192.168.101.41@tcp,192.168.102.41@tcp1 --param mgsnode=192.168.101.42@tcp,192.168.102.42@tcp1 /dev/vdb</p> <p>This way, the old format is used in CONFIGS/mountdata.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21329" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21329</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8311" title="Target does not mount with the new mgsnode parameter format in case of multirail configuration" class="issue-link" data-issue-key="LU-8311"><del>LU-8311</del></a> mount: fix lmd_parse() to parse colon as NID delimiter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e74f6c0414f6ef538877e9fbca8540c4be3f2699</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21329/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21329/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8311" title="Target does not mount with the new mgsnode parameter format in case of multirail configuration" class="issue-link" data-issue-key="LU-8311"><del>LU-8311</del></a> mount: fix lmd_parse() to parse colon as NID delimiter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2458067d8d55173ad68caac8c0460d46bf8106a1</p> <p>Landed for 2.9</p> <p>The <tt>mount.lustre</tt> and <tt>mkfs.lustre</tt> man pages need to be updated to include better examples of how NIDs can be specified.</p> <ul class="alternate" type="square"> <li><tt>mount.lustre</tt> explains <tt>mgsspec</tt>, <tt>mgsnode</tt>, and <tt>mgsnid</tt> but does not provide an example of an MGS with a failover node and multiple NIDs.</li> <li><tt>mkfs.lustre</tt> does not explain <tt>nid</tt>/<tt>NID</tt> and also does not include anything but trivial examples.</li> </ul> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23355" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23355</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8311" title="Target does not mount with the new mgsnode parameter format in case of multirail configuration" class="issue-link" data-issue-key="LU-8311"><del>LU-8311</del></a> doc: add NIDs examples to mkfs.lustre and mount.lustre<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7bfd77d5b7a34bf8dba529276d6516e40d59cc3d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23355/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23355/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8311" title="Target does not mount with the new mgsnode parameter format in case of multirail configuration" class="issue-link" data-issue-key="LU-8311"><del>LU-8311</del></a> doc: add NIDs examples to mkfs.lustre and mount.lustre<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fd488bacbf31b623815cf86ca21e5d6c888c068e</p> <p>Landed for 2.9</p> <p>We are running into something very similar to this - not sure if its related or something different. Lots of detail in a thread on the mailing list - here is a link to one of the latest posts. </p> <p><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-January/014154.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-January/014154.html</a></p> <p>The summary of our situation is that our LFS was formatted originally using 2.8 but we have since upgraded to 2.9.51 We are using a JBOB with server pairs for failover and are using ZFS as the backend. All servers are dual-homed on both ethernet and IB. MDT and OST failover works fine. MGS failover doesn't work if we have both ethernet and IB but does if only have ethernet NID's. We have build our own lustre server RPM's using a "git checkout 2.9.51" and zfs 0.6.5.8-1. I've verified that commit 2458067d8d55173ad68caac8c0460d46bf8106a1 is in the git log. Any help would be much appreciated. </p> <p>It is important to note that the 2.9.51 code is a development tag and as such has no expectation of being tested or supported. Development tags may contain protocol changes and experimental code, so unless you are using this only for testing the stability of the development branch, I would suggest to go back to 2.9.0. </p> <p>Good to know - will do. </p> <p>I just uploaded a couple of debug logs. Both were taken with while mounting an OST on one of our OSS's. One was while we were configured only for ethernet.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> tunefs.lustre \ --verbose \ --writeconf \ --erase-param \ --mgsnode=192.52.98.30@tcp0 \ --mgsnode=192.52.98.31@tcp0 \ --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0 \ --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0 \ $pool/ost-fsl </pre> </div></div> <p>The other was while configured with both IB and ethernet.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> tunefs.lustre \ --verbose \ --writeconf \ --erase-param \ --mgsnode=192.52.98.30@tcp0,10.148.0.30@o2ib0 \ --mgsnode=192.52.98.31@tcp0,10.148.0.31@o2ib0 \ --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \ --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \ $pool/ost-fsl </pre> </div></div> <p>Hi Darby,</p> <p>From the debug log in debug.log.ib_and_eth, I didn't see any MGS NID was not parsed by lmd_parse(). So, it's not the same issue as this one.<br/> It looks like there is a defect in lustre_start_mgc() introduced by patch <a href="https://review.whamcloud.com/7509" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7509</a> for <a href="https://jira.whamcloud.com/browse/LU-3829" title="MDT mount fails if mkfs.lustre is run with multiple mgsnode arguments on MDSs where MGS is not running" class="issue-link" data-issue-key="LU-3829"><del>LU-3829</del></a>. And a similar issue was reported in <a href="https://jira.whamcloud.com/browse/LU-8397" title="take comma as separator of mgsnode&#39;s list" class="issue-link" data-issue-key="LU-8397"><del>LU-8397</del></a>. I'll add comments in that ticket.</p> <p>Thanks a lot for the info.  If you need any more data from me, I'd be glad to post that - either to this LU or one those others.  </p> <p>I'd like to try reverting that patch from the 2.9 release and see if it fixes our issue.  Please let me know if you think that's worthwhile and, if so, which LU I should post the info to.  </p> <p>Thank you Darby. The issue will be resolved in <a href="https://jira.whamcloud.com/browse/LU-8397" title="take comma as separator of mgsnode&#39;s list" class="issue-link" data-issue-key="LU-8397"><del>LU-8397</del></a>.</p> Related LU-4334 LU-5690 Development Rank 1|hzyf5z: Rank (Obsolete) 9223372036854775807 Severity