[LU-4966] MGS target registration should use proper UUID

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: llnl
• Assignee: WC Triage
• Reporter: Niu Yawei (Inactive)

Description

If some server registered successfully on MGS, but it got an error registration reply (MGS revoking config locks timeout or other networking problems), then the server will always get -EADDRINUSE error when it try to register next time, because the server index has been occupied on MGS in the first registration.

Current solution for above situation is to use writeconf option to force registration.

We need to get this improved and make MGS able to handle this gracefully.

Issue Links

• duplicates: LU-3458 OST not able to register at MGS with predefined index.
• duplicates: LU-17716 add 'tunefs.lustre --replace' to allow OST to be skip MGS registration
• is related to: LU-18835 'chown' in nodemap sets UID/GID to '65534' on OST objects
• is related to: LU-18857 tunable parameter to allow/deny new MDT/OST registration with MGS
• is related to: LU-1257 OST registration snafu
• is related to: LU-9838 target registration mount fails with -108 but then succeeds if retried immediately
• is related to: LU-15112 attempt to register an OST with duplicated index should fail but it does not
• is related to: LU-14928 Allow MD target re-registered after writeconf
• is related to: LU-17240 change test-framework to format and mount targets in parallel

Comments

[Sebastien Piechurski]

Was this issue somehow adressed in latest versions ?
We are still seeing this problem when installing configurations with large amount of OSTs (or when modifiying it after a writeconf) and have to take care manually of it by starting OSTs one after another.

[Andreas Dilger]

There is space in the last_rcvd file to store the UUID, but that has the potential problem that this file may be deleted if there are problems with recovery.

As for the OST detection in the connection cide, it would be possible to store the target type in the last byte of the UUID or similar (e.g. the ASCII "O" or "M") and still make the rest of the UUID random.

[Li Wei (Inactive)]

Niu's idea makes sense to me. I spent some time experimenting with it.

A real UUID (e.g., "a53bc5ba-687b-4091-fb0b-61489785f247") could easily be generated by back-end-independent mkfs.lustre code and stored in a back-end-specific way (e.g., in ldiskfs "mountdata" or as a ZFS dataset property). (ZFS has pool IDs, but those are pool properties and are only 64-bit.)

Current master code always passes empty strings in mti_uuid. It would be nice if real UUIDs could be packed into that field. However, experiments showed:

• MDT OSPs for OSTs would send MDS_CONNECTs to OSTs, because client_obd_setup() depends on "OST" in UUIDs to determine whether an OSP is for an OST or an MDT.
• Clients would not be able to connect to MDTs, because mgs would put fake UUIDs (e.g., "lustre-MDT0000_UUID") into MDT logs but real UUIDs into the client log.

A possible solution is:

• Start generating real UUIDs for any newly formatted targets.
• Send real UUIDs via mti_uuid.
• MGT checks and stores real UUIDs somewhere, but keeps putting fake UUIDs into logs.
• Newly formatted targets must talk with new mgs code.

[Andreas Dilger]

Alex, I think Niu was asking for ideas on how this might best be fixed.

[Alex Zhuravlev]

this issue was mentioned by Chris in LU-1257 and at the moment it's not clear whether this specific issue is a major one for LLNL. it'll be hard to get this fixed in 2.6 due to amount of changes ?

[Niu Yawei (Inactive)]

Andreas/Alex, any suggestions? Thanks.

[Niu Yawei (Inactive)]

I think if MGS save server UUID along with the server index, the it can tell if the registration (acquire for an occupied index) come from same server.
And looks MGS now keeps server index bitmap in memory only, it needs be saved in disk as well.