[LU-13306] allow clients to accept mgs_nidtbl_entry with IPv6 NIDs

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Labels: IPv6
• Assignee: James A Simmons
• Reporter: Andreas Dilger

Description

Currently, if a client receives an mgs_nidtbl_entry with mne_nid_type != 0 or mne_nid_size != sizeof(lnet_nid_t)=8 the client will stop processing all later records, essentially cutting the client off from OST/MDT IR logs that have even one IPv6 addresses configured.

To add IPv6 NID support, but preserve compatibility with older clients, there should be a new connect flag sent to the MGS at connect time:

#define OBD_CONNECT2_MNE_TYPE         0x1000000ULL /* mne_nid_type flags */

that should be saved in the client export on the MGS and used by mgs_nidtbl_read() to decide what type of NIDs should be sent to the client. The client should also send the flags that it understands MNE_NID_SUPPORTED = (MNE_NID_TYPE_* | (MNR_REC_FLAG_* << 8)) in struct obd_connect_data.padding0 renamed to ocd_mne_type so that the MGS can also avoid sending future record types to a client that it doesn't understand. The MGS reply should mask out any flags in ocd_mne_type that it doesn't support itself.  If a client export does not have OBD_CONNECT2_MNE_TYPE set, the MGS should only send compatible lnet_nid_t records with mne_nid_type=0 and mne_nid_size=sizeof(lnet_nid_t)=8.

To allow future expansion, the mne_rec_type field should be treated as a bitmask of features present in the mgs_nidtbl_entry instead of a enumerated type. The first bit set in mne_nid_type=MNE_NID_TYPE_RECORD=0x01 would be used for expanding the record size, with the larger per-NID records as follows, with mne_nid_size=24, and a per-record mnr_rec_flags=MNR_REC_FLAG_NIDv6=0x01 to allow positively identifying the record content rather than depending on "all zero" fields to detect the difference:

struct mne_nid_record {         /* when MNE_NID_TYPE_RECORD is set */
        __u8                    mnr_rec_flags;  /* MNR_REC_FLAG_* flags */
        __u8                    mnr_padding1;
        __u16                   mnr_padding2;
        struct lnet_nid_v6      mnr_nid_v6;
        /* other fields depending on mne_nid_type, mne_nid_size, mnr_rec_flags */
};

struct mgs_nidtbl_entry {
        __u64           mne_version;    /* table version of this entry */
        __u32           mne_instance;   /* target instance # */
        __u32           mne_index;      /* target index */
        __u32           mne_length;     /* length of this entry - by bytes */
        __u8            mne_type;       /* target type LDD_F_SV_TYPE_OST/MDT */
        __u8            mne_nid_type;   /* MNE_NID_TYPE_* flags */
        __u8            mne_nid_size;   /* size of each NID, by bytes */
        __u8            mne_nid_count;  /* # of NIDs in buffer */
        union {         /* don't walk the union array directly, use mne_nid_size */
                lnet_nid_t             mne_nids[];     /* when mne_nid_type == 0 */
                struct mnr_nid_record mne_records[];  /* if MNE_NID_TYPE_RECORD */
        };
};

The 4 extra bytes of fields at the start plus the 20-byte mnr_nid_v6 field keep all records with 64-bit size and alignment. Since lnet_nid_v6 can also encapsulate an IPv4 NID (based on .nid_type) there is no need to handle lnet_nid_t directly within mnr_nid_record. However, MNR_REC_FLAG_NIDv6 should be verified in case there is some other kind of NID in the record.

When processing mne_nids or mne_records, it should be using mne_nid_size to walk the records, rather than having a loop that processes mne_nids[i] as an array, otherwise this will break if mne_nid_size changes in the future (ignoring any potential changes from LU-10360):

                void *ptr;

                /* iterate all nids to find one UUID by NID */
                rc = -ENOENT;

                for (i = 0, ptr = entry->mne_records; i < entry->mne_nid_count; i++, ptr += entry->mne_nid_size) {
                        lnet_nid_v6 nid_v6;

                        if (entry->mne_nid_type & MNE_NID_TYPE_RECORD) {
                                struct mnr_nid_record *rec = ptr;
  
                                if (!(rec->mnr_rec_flags & MNR_REC_FLAG_NIDv6))
                                        continue;
                                nid_v6 = rec->mnr_nid_v6;
                        } else {
                                lnet_nid2nidv6(*(lnet_nid_t *)ptr, &nid_v6);
                        }
            
                        rc = client_import_find_conn(obd->u.cli.cl_import, nid_v6,
                                                     (struct obd_uuid *)uuid);
                        if (rc == 0)
                                break;
                }

Issue Links

• is related to: LU-17060 conf-sanity test_21c: Unable to start OST2 - MGS refuses to allow this server to start
• is related to: LU-18049 ost-pools test_25, sanity-sec test_31: crash in ext4_htree_store_dirent kmalloc
• is related to: LU-17962 conf-sanity test_32a: failed with replace_nids operation already in progress
• is related to: LU-16722 MGS config log restructuring
• is related to: LU-16823 add LNet and OBD connect flags for IPv6 peers
• is related to: LU-10360 use Imperative Recovery logs for client->MDT/OST connections
• is related to: LU-19 imperative recovery
• is related to: LU-10391 LNET: Support IPv6

Comments

[Peter Jones]

All patches seem to be merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50750/
Subject: LU-13306 mgc: handle large NID formats
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e4d2d4ff74a103e4288d7581b40020eaff22b139

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52053/
Subject: LU-13306 mgs: support large NID for mgs_write_log_osc_to_lov
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 576928b2cd7dcdb6fcb6000ef593ecfb03f66500

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52053
Subject: LU-13306 mgs: support large NID for mgs_write_log_osc_to_lov
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e28695e90051d33bc5129d3512cdd9e01297f4d8

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50896/
Subject: LU-13306 mgs: use large NIDS in the nid table on the MGS
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c0cb747ebe95d3fefa2a70c09c2ae0c4fa873b47

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51185
Subject: LU-13306 test: base code
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ba68cc5f0b71cccc60b3f240cbcacba82796a29c

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50896
Subject: LU-13306 mgs: use large NIDS in the nid table on the MGS
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6ea3f9538d8ce242f87d96731e2ee4d0900c3983

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50750
Subject: LU-13306 mgc: handle large NIDs gracefully
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 86d7f923d7caec756f0354d774e511482c138f50