[LU-4749] ZFS-backed OST mkfs.lustre --servicenode does not correctly add failover_nids

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.2, Lustre 2.7.0
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Type: Bug
• Priority: Blocker
• Reporter: Anthony Alba
• Assignee: Li Wei (Inactive)
• Resolution: Fixed
• Labels: prz, zfs

Environment

CentOS 6.4, ZFS 0.6.2

• Epic/Theme: ZFS
• Severity: 3

Description

When creating ZFS-backed OSTs using the --servicenode syntax, only one failover nids is stored.

mkfs.lustre --ost --index=1 --fsname=saturn --backfstype=zfs --mgsnode=192.168.122.73@tcp --servicenode=192.168.122.76@tcp --servicenode=192.168.122.78@tcp lsrv3/saturn-ost1

1. tunefs.lustre --print lsrv3/saturn-ost1
   checking for existing Lustre data: found

Read previous values:
Target: saturn-OST0001
Index: 1
Lustre FS: saturn
Mount type: zfs
Flags: 0x1002
(OST no_primnode )
Persistent mount opts:
Parameters: failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp

Permanent disk data:
Target: saturn-OST0001
Index: 1
Lustre FS: saturn
Mount type: zfs
Flags: 0x1002
(OST no_primnode )
Persistent mount opts:
Parameters: failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp

On MGS:

1. lctl get_param osp.saturn-OST0001-osc-MDT0000.import
   osp.saturn-OST0001-osc-MDT0000.import=
   import:
   name: saturn-OST0001-osc-MDT0000
   target: saturn-OST0001_UUID
   state: FULL
   instance: 1
   connect_flags: [lov_index, unused, version, request_portal, adaptive_timeouts, lru_resize, fid_is_enabled, skip_orphan, full20, lvb_type]
   import_flags: [replayable, pingable]
   connection:
   failover_nids: [192.168.122.78@tcp]
   current_connection: 192.168.122.78@tcp

For a ldiskfs-backed OST, two NIDs are stored:

1. mkfs.lustre --ost --index=0 --fsname=saturn --servicenode=192.168.122.76@tcp --mgsnode=192.168.122.73@tcp --reformat /dev/vdb

1. tunefs.lustre --print /dev/vdb
   checking for existing Lustre data: found
   Reading CONFIGS/mountdata

Read previous values:
Target: saturn-OST0000
Index: 0
Lustre FS: saturn
Mount type: ldiskfs
Flags: 0x1002
(OST no_primnode )
Persistent mount opts: errors=remount-ro
Parameters: failover.node=192.168.122.76@tcp failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp

Permanent disk data:
Target: saturn-OST0000
Index: 0
Lustre FS: saturn
Mount type: ldiskfs
Flags: 0x1002
(OST no_primnode )
Persistent mount opts: errors=remount-ro
Parameters: failover.node=192.168.122.76@tcp failover.node=192.168.122.78@tcp mgsnode=192.168.122.73@tcp

exiting before disk write.

1. lctl get_param osp.saturn-OST0000-osc-MDT0000.import
   osp.saturn-OST0000-osc-MDT0000.import=
   import:
   name: saturn-OST0000-osc-MDT0000
   target: saturn-OST0000_UUID
   state: FULL
   instance: 2
   connect_flags: [lov_index, unused, version, request_portal, adaptive_timeouts, lru_resize, fid_is_enabled, skip_orphan, full20, lvb_type]
   import_flags: [replayable, pingable]
   connection:
   failover_nids: [192.168.122.76@tcp, 192.168.122.78@tcp]
   current_connection: 192.168.122.76@tcp

Issue Links

• is related to: LU-4334 With ZFS can only declare a single mg...

Comments

[Anthony Alba]

For the ldiskfs I omitted adding the failover NID:

1. tunefs.lustre --erase-params --servicenode=192.168.122.76@tcp --servicenode=192.168.122.78@tcp --mgsnode=192.168.122.73@tcp /dev/vdb

[Anthony Alba]

1. A second oddity: I think --mgsnode=ABCD --mgsnode=XYZW also doesn't work on 2.4.2/ZFS-backed OSTs. The 2nd mgsnode overrides the first. For LDISKFS-backed OSTs it seems to work.

2. Does the syntax --mgsnode=Pri_NID:Sec_NID work for mkfs.lustre or should one be using
--mgsnode=Pri_NID --mgsnode=Sec_NID

[Jodi Levi (Inactive)]

http://review.whamcloud.com/11161 will fix this issue

[Isaac Huang (Inactive)]

This looks like a duplicate of LU-4334. Both tunefs.lustre and mkfs.lustre are built from the same sources: mkfs_lustre.c mount_utils.c mount_utils.h.

[Li Wei (Inactive)]

I was waiting for the LU-4334 patch to fix this, but that patch turned out to insufficient. Current mgs cannot handle colons in failover.node values. I'll submit my patch soon.

[Li Wei (Inactive)]

http://review.whamcloud.com/11956

[Jodi Levi (Inactive)]

Patch landed to Master

[Bob Glossman (Inactive)]

backport to b2_5:
http://review.whamcloud.com/12196

[Oleg Drokin]

I think this patch causes failures in LU-5706, e.g.: https://testing.hpdd.intel.com/test_sessions/6c3961a2-4aa8-11e4-95b1-5254006e85c2

Once this was cherrypicked to b2_5 as a separate patch, it started to hit.
Might be it was a fluke, but reverting it made the problem disappear.
http://review.whamcloud.com/#/c/12166/ vs http://review.whamcloud.com/#/c/12183/

[Oleg Drokin]

Also looking in maloo results, it's really end of Sptember where these problems started to appear, and before that all failures were in 2013.

So i think chances are high this is the culprit.

[Andreas Dilger]

Reopen due to potential problems with the patch.

[Andreas Dilger]

It looks like there is some garbage being written into the ZFS properties. From the test log output of https://testing.hpdd.intel.com/test_sets/a33da7e2-4a9b-11e4-adcb-5254006e85c2

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 mgsnode=10.1.5.243@tcp failover.node=��6

Writing lustre-ost1/ost1 properties
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
  lustre:sys.timeout=20
  lustre:mgsnode=10.1.5.243@tcp
  lustre:failover.node=��6

[Li Wei (Inactive)]

This is not the cause of LU-5706; please see my comments there.

[Li Wei (Inactive)]

I think this should be either closed or left open for Bob's b2_5 port. Removed the link to LU-5706.