[LU-8397] take comma as separator of mgsnode's list

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Severity: 3
• Assignee: Emoly Liu
• Reporter: Vladimir Saveliev

Description

mgsnode's list of nids is allowed to use comma as separator (see 36.15.3 of Lustre Operations Manual), however lustre_start_mgc() assumes that only colon can separate nids. Due to that nids after comma do not get into list of import's connections and those nids do not get involved.

Issue Links

• is related to: LU-10162 mount command does not appear to be using the failover MGS/second MGS
• is related to: LU-3829 MDT mount fails if mkfs.lustre is run with multiple mgsnode arguments on MDSs where MGS is not running

Comments

[Jian Yu]

Patch landed to master branch for Lustre 2.10.0.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/21308/
Subject: LU-8397 mgc: add comma-separated nids for primary MGS
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 672e182049e1a636d8591748764950f99c4b71d9

[Jian Yu]

Thank you very much, Darby. I'll push the patch to be ready to land.

[Darby Vicker]

I just tested 2.9.0 with this patch and it does indeed fix our issues.  I brought up our filesystem while configured on both TCP and IB, mounted both TCP and IB clients and failed over the MDS/MGS to the secondary.  The OST's and the clients all pick up the secondary MGS now.  Info from one of our OSS's is below - this is after the failover.  Thank you very much!  

 

[root@hpfs-fsl-oss01 ~]# cat /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import 
import:
    name: MGC192.52.98.30@tcp
    target: MGS
    state: FULL
    connect_flags: [ version, adaptive_timeouts, mds_mds_connection, full20, imp_recov, bulk_mbits ]
    connect_data:
       flags: 0x2000011005000020
       instance: 0
       target_version: 2.9.0.0
    import_flags: [ pingable, connect_tried ]
    connection:
       failover_nids: [ 192.52.98.31@tcp, 192.52.98.30@tcp ]
       current_connection: 192.52.98.31@tcp
       connection_attempts: 3
       generation: 2
       in-progress_invalidations: 0
[root@hpfs-fsl-oss01 ~]# zfs get all | grep mgs
oss01-0/ost-fsl             lustre:mgsnode        192.52.98.30@tcp,10.148.0.30@o2ib:192.52.98.31@tcp,10.148.0.31@o2ib  local
[root@hpfs-fsl-oss01 ~]# 

[Jian Yu]

Hi Darby,
I just updated patch https://review.whamcloud.com/21308 on master branch to fix the issue in my previous comment. The patch can be applied to Lustre 2.9.0 release directly. Could you please try it? Thank you.

[Darby Vicker]

I'm sorry - it turns out I was wrong when I said reverting this patch fixed our situation.  I think what happened is that I forgot to stop lnet before I uninstalled stock 2.9.0.  I then installed and tested the patched version of 2.9.0 - but the lnet kernel modules from stock 2.9.0 stayed loaded.  So I was using some weird combination of a patched and stock version of 2.9.0.  I've repeated the tests using a version of 2.9.0 with patch 7905 reverted and the situation is actually worse - with both single rail (tcp only) or dual rail (tcp and ib) I only see one entry for failover_nids in /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import.  I can repeat the tests and dump debug logs if that would help.  

[Jian Yu]

Thank you very much Darby for the confirmation.

[Darby Vicker]

This evening I compiled the 2.9.0 release with patch 7905 reverted.  I can confirm that this did allow me to failover our OST's properly.  Once the OST were mounted with the NIDS shown in the comment above we see both TCP NIDS in /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import and the MGC connects with the secondary MDS/MGS when we failover.