[LU-16096] recovery: handle compatibility during upgrade for new replay data format

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.16.0
• Labels: LTS15 statahead
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

As batched RPC protocol will change the disk format of the client reply data "REPLY_DATA" for recovery, thus we need to handle compatibility during upgrade carefully for this new replay data format.

The new format is introduced in https://review.whamcloud.com/#/c/46799/.

The new format is as follow:

struct lsd_reply_data
{ 
__u64 lrd_transno; /* transaction number */
__u64 lrd_xid; /* transmission id */
__u64 lrd_data; /* per-operation data */
__u32 lrd_result; /* request result */
__u32 lrd_client_gen; /* client generation */
+__u32 lrd_batch_idx; /* sub request index in a batched RPC */
+__u32 lrd_padding[7]; /* unused fields. */ 
};

The proposed solution is as follows:

Add several flags in the magic number field of the reply data header:

LRH_MAGIC_V1: 0xbdabda01 - the magic number of the old format for client reply data.

LRH_MAGIC: 0xbdabda02 - the magic number of the new format for the client reply data.

LRH_FLAG_BACKUP_DONE: 0x00000004 - indicate the target has finished to backup the "REPLY_DATA" with old format.

 

During the target setup, it will initialize the reply data in @tgt_init()->tgt_reply_data_init().

1. if found that the "REPLY_DATA" is old format (according to the magic number in the reply data header "LRH_MAGIC"),  the target starts to backup the "REPLY_DATA" file into the file "REPLY_DATA_BAK".
2. After finished the backup, the target will change the magic number field of the reply data header with LRH_MAGIC_V1 | LRH_FLAG_BACKUP_DONE, and sync the magic flag change into the persistent storage.
3. The target starts to convert the old format reply data from the backup file "REPLY_DATA_BAK" into the original reply data file "REPLY_DATA".
4. After finished the conversion, the target changes the magic number @lrh_magic of the reply data header with LRH_MAGIC and @lrh_reply_size with new format, and sync the change to the disk. After that delete the backup file "REPLY_DATA_BAK".
5. After that, the target starts the recovery. processing as normal with the new format reply data.

 

Issue Links

• is related to: LU-15975 Statahead_V1 Features
• is related to: LU-14139 batched statahead processing

Comments

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50636/
Subject: LU-16096 target: use lsd_reply_data_v1 format by default
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5321d510878f8893a49778025c6981e46a66cdff

[Andreas Dilger]

James,
the lack of compatibility is causing the clean-downgrade and clean-downgrade-zfs tests to fail with "invalid header in reply_data":
https://testing.whamcloud.com/test_sets/26ed0085-1e4b-4c6d-acc7-ca63a5775066

LustreError: 74648:0:(tgt_lastrcvd.c:2070:tgt_reply_data_init()) lustre-MDT0000: invalid header in reply_data
LustreError: 74648:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-22)
 LustreError: 74648:0:(obd_config.c:2029:class_config_llog_handler()) MGC10.240.26.9@tcp: cfg command failed: rc = -22
Lustre:    cmd=cf003 0:lustre-MDT0000  1:lustre-MDT0000_UUID  2:0  3:lustre-MDT0000-mdtlov  4:f  

LustreError: 15b-f: MGC10.240.26.9@tcp: Configuration from log lustre-MDT0000 failed from MGS -22. Check client and MGS are on compatible version.
LustreError: 74442:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -22
LustreError: 74442:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -22
LustreError: 74442:0:(obd_config.c:827:class_cleanup()) Device 5 not setup
LustreError: 74521:0:(ldlm_lockd.c:2500:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1686822829 with bad export cookie 6862479036220380163
LustreError: 166-1: MGC10.240.26.9@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: server umount lustre-MDT0000 complete
LustreError: 74442:0:(super25.c:176:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -22

The patch patch: https://review.whamcloud.com/50636 "LU-16096 target: use lsd_reply_data_v1 format by default" still needs to land so that upgrade/downgrade continues to work.

[James A Simmons]

Is this work complete?

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49939/
Subject: LU-16096 tgt: improve messages for reply_data
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b2f05051c4239e845434ea9e183d889e74a5db57

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50636
Subject: LU-16096 target: use lsd_reply_data_v1 format by default
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4b4c1ea61bcc2029b4db9e4ca106d42eac2257a4