https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4749 Lustre <p>When creating ZFS-backed OSTs using the --servicenode syntax, only one failover nids is stored.</p> <p>mkfs.lustre --ost --index=1 --fsname=saturn --backfstype=zfs --mgsnode=192.168.122.73@tcp --servicenode=192.168.122.76@tcp --servicenode=192.168.122.78@tcp lsrv3/saturn-ost1</p> <ol> <li>tunefs.lustre --print lsrv3/saturn-ost1<br/> checking for existing Lustre data: found</li> </ol> <p> Read previous values:<br/> Target: saturn-OST0001<br/> Index: 1<br/> Lustre FS: saturn<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp</p> <p> Permanent disk data:<br/> Target: saturn-OST0001<br/> Index: 1<br/> Lustre FS: saturn<br/> Mount type: zfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: <br/> Parameters: failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp</p> <p>On MGS:</p> <ol> <li>lctl get_param osp.saturn-OST0001-osc-MDT0000.import<br/> osp.saturn-OST0001-osc-MDT0000.import=<br/> import:<br/> name: saturn-OST0001-osc-MDT0000<br/> target: saturn-OST0001_UUID<br/> state: FULL<br/> instance: 1<br/> connect_flags: <span class="error">&#91;lov_index, unused, version, request_portal, adaptive_timeouts, lru_resize, fid_is_enabled, skip_orphan, full20, lvb_type&#93;</span><br/> import_flags: <span class="error">&#91;replayable, pingable&#93;</span><br/> connection:<br/> failover_nids: <span class="error">&#91;192.168.122.78@tcp&#93;</span><br/> current_connection: 192.168.122.78@tcp</li> </ol> <p>For a ldiskfs-backed OST, two NIDs are stored:</p> <ol> <li>mkfs.lustre --ost --index=0 --fsname=saturn --servicenode=192.168.122.76@tcp --mgsnode=192.168.122.73@tcp --reformat /dev/vdb</li> </ol> <ol> <li>tunefs.lustre --print /dev/vdb<br/> checking for existing Lustre data: found<br/> Reading CONFIGS/mountdata</li> </ol> <p> Read previous values:<br/> Target: saturn-OST0000<br/> Index: 0<br/> Lustre FS: saturn<br/> Mount type: ldiskfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: errors=remount-ro<br/> Parameters: failover.node=192.168.122.76@tcp failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp</p> <p> Permanent disk data:<br/> Target: saturn-OST0000<br/> Index: 0<br/> Lustre FS: saturn<br/> Mount type: ldiskfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: errors=remount-ro<br/> Parameters: failover.node=192.168.122.76@tcp failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp</p> <p>exiting before disk write.</p> <ol> <li>lctl get_param osp.saturn-OST0000-osc-MDT0000.import<br/> osp.saturn-OST0000-osc-MDT0000.import=<br/> import:<br/> name: saturn-OST0000-osc-MDT0000<br/> target: saturn-OST0000_UUID<br/> state: FULL<br/> instance: 2<br/> connect_flags: <span class="error">&#91;lov_index, unused, version, request_portal, adaptive_timeouts, lru_resize, fid_is_enabled, skip_orphan, full20, lvb_type&#93;</span><br/> import_flags: <span class="error">&#91;replayable, pingable&#93;</span><br/> connection:<br/> failover_nids: <span class="error">&#91;192.168.122.76@tcp, 192.168.122.78@tcp&#93;</span><br/> current_connection: 192.168.122.76@tcp</li> </ol> CentOS 6.4, ZFS 0.6.2 LU-4749

ZFS-backed OST mkfs.lustre --servicenode does not correctly add failover_nids

Bug Blocker Resolved Fixed Li Wei Anthony Alba prz zfs Tue, 11 Mar 2014 14:03:20 +0000 Mon, 27 Apr 2015 20:17:51 +0000 Thu, 9 Oct 2014 16:24:43 +0000 Lustre 2.4.2 Lustre 2.7.0 Lustre 2.7.0 Lustre 2.5.4 0 8 <p>For the ldiskfs I omitted adding the failover NID:</p> <ol> <li>tunefs.lustre --erase-params --servicenode=192.168.122.76@tcp --servicenode=192.168.122.78@tcp --mgsnode=192.168.122.73@tcp /dev/vdb</li> </ol> <p>1. A second oddity: I think --mgsnode=ABCD --mgsnode=XYZW also doesn't work on 2.4.2/ZFS-backed OSTs. The 2nd mgsnode overrides the first. For LDISKFS-backed OSTs it seems to work.</p> <p>2. Does the syntax --mgsnode=Pri_NID:Sec_NID work for mkfs.lustre or should one be using<br/> --mgsnode=Pri_NID --mgsnode=Sec_NID</p> <p><a href="http://review.whamcloud.com/11161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11161</a> will fix this issue</p> <p>This looks like a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4334" title="With ZFS can only declare a single mgsnode for MDT or OST" class="issue-link" data-issue-key="LU-4334"><del>LU-4334</del></a>. Both tunefs.lustre and mkfs.lustre are built from the same sources: mkfs_lustre.c mount_utils.c mount_utils.h.</p> <p>I was waiting for the <a href="https://jira.whamcloud.com/browse/LU-4334" title="With ZFS can only declare a single mgsnode for MDT or OST" class="issue-link" data-issue-key="LU-4334"><del>LU-4334</del></a> patch to fix this, but that patch turned out to insufficient. Current mgs cannot handle colons in failover.node values. I'll submit my patch soon.</p> <p><a href="http://review.whamcloud.com/11956" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11956</a></p> <p>Patch landed to Master</p> <p>backport to b2_5:<br/> <a href="http://review.whamcloud.com/12196" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12196</a></p> <p>I think this patch causes failures in <a href="https://jira.whamcloud.com/browse/LU-5706" title="conf-sanity test_57a: @@@@@@ FAIL: OST registration from failnode should fail " class="issue-link" data-issue-key="LU-5706"><del>LU-5706</del></a>, e.g.: <a href="https://testing.hpdd.intel.com/test_sessions/6c3961a2-4aa8-11e4-95b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/6c3961a2-4aa8-11e4-95b1-5254006e85c2</a></p> <p>Once this was cherrypicked to b2_5 as a separate patch, it started to hit.<br/> Might be it was a fluke, but reverting it made the problem disappear.<br/> <a href="http://review.whamcloud.com/#/c/12166/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12166/</a> vs <a href="http://review.whamcloud.com/#/c/12183/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12183/</a></p> <p>Also looking in maloo results, it's really end of Sptember where these problems started to appear, and before that all failures were in 2013.</p> <p>So i think chances are high this is the culprit.</p> <p>Reopen due to potential problems with the patch.</p> <p>It looks like there is some garbage being written into the ZFS properties. From the test log output of <a href="https://testing.hpdd.intel.com/test_sets/a33da7e2-4a9b-11e4-adcb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a33da7e2-4a9b-11e4-adcb-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 mgsnode=10.1.5.243@tcp failover.node=����6 Writing lustre-ost1/ost1 properties lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 lustre:sys.timeout=20 lustre:mgsnode=10.1.5.243@tcp lustre:failover.node=����6 </pre> </div></div> <p>This is not the cause of <a href="https://jira.whamcloud.com/browse/LU-5706" title="conf-sanity test_57a: @@@@@@ FAIL: OST registration from failnode should fail " class="issue-link" data-issue-key="LU-5706"><del>LU-5706</del></a>; please see my comments there.</p> <p>I think this should be either closed or left open for Bob's b2_5 port. Removed the link to <a href="https://jira.whamcloud.com/browse/LU-5706" title="conf-sanity test_57a: @@@@@@ FAIL: OST registration from failnode should fail " class="issue-link" data-issue-key="LU-5706"><del>LU-5706</del></a>.</p> Related LU-4334 Development Epic/Theme ZFS Rank 1|hzwhfr: Rank (Obsolete) 13075 Severity