[LU-15935] MDT mount fails with "duplicate generation for client export" during failover

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.14.0, Lustre 2.12.8, Lustre 2.12.9
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.3
• Type: Bug
• Priority: Critical
• Reporter: Andreas Dilger
• Assignee: Etienne Aujames
• Resolution: Fixed
• Severity: 3

Description

Mounting the MDT during recovery immediately fails with:

(tgt_lastrcvd.c:1575:tgt_clients_data_init()) fs2-MDT000f: duplicate export for client generation 17
(obd_config.c:559:class_setup()) setup fs2-MDT000f failed (-114)
(obd_config.c:1836:class_config_llog_handler()) MGC10.11.12.142@o2ib: cfg command failed: rc = -114
Lustre:    cmd=cf003 0:fs2-MDT000f  1:fs2-MDT000f_UUID  2:15  3:fs2-MDT000f-mdtlov  4:f  
15c-8: MGC10.11.12.142@o2ib: The configuration from log 'fs2-MDT000f' failed (-114).
    This may be the result of communication errors between this node and the MGS, a bad
    configuration, or other errors. See the syslog for more information.
obd_mount_server.c:1408:server_start_targets()) failed to start server fs2-MDT000f: -114
obd_mount_server.c:2052:server_fill_super()) Unable to start targets: -114

There appear to be duplicate client recovery records in the MDT's last_rcvd file that are preventing it from creating all of the client exports in tgt_clients_data_init(). It doesn't have anything to do with the MGS, despite the later messages. Removing the last_rcvd file allowed the MDT to be mounted, as likely mounting with "-o abort_recov" would have done.

Issue Links

• is duplicated by: LU-7794 tgt_clients_data_init()) soaked-MDT00...
• is related to: LU-6468 Interop 2.5.3<->master conf-sanity te...

Comments

[Andreas Dilger]

Rather than failing the mount in this case, it makes sense to evict the client(s) that have the duplicate slots and continuing the mount, since there is no other course of action at this point anyway, and evicting a single client is better than aborting recovery (or deleting the last_rcvd file) and evicting all of the clients. That would allow the system to be mounted instead of needing manual intervention to finish the recovery.

[Etienne Aujames]

Hi Andreas,

We run into this at the CEA with, 8000+ clients. I have not direct access to this machine, but the CEA will try to get some logs.

Can you elaborate how it happens, and how can we reproduce this issue?

[Andreas Dilger]

Etienne, sorry but I don't know why this happens, but I think it would be reasonable to either ignore this error or at most evict the client that has the duplicate export, possibly printing a bit more details about the client NID and related info to see why it happens.

[Etienne Aujames]

I successfully reproduced an issue that could lead to "duplicate generation..." with abort_recovery:

Here are the steps to reproduce it:

1. mount 2 clients

   last_rcvd:
     uuid: lustrefs-MDT0000_UUID
     feature_compat: 0x8
     feature_incompat: 0x61c              <--- (0x400/OBD_INCOMPAT_MULTI_RPCS is set)
     feature_rocompat: 0x1
     last_transaction: 635655159808
     target_index: 0
     mount_count: 159
     client_area_start: 8192
     client_area_size: 128
   
     c57cde24-ed24-8705-9890-6d7bab261c25:
       generation: 2
       last_transaction: 0
       last_xid: 0
       last_result: 0
       last_data: 0
   
     160e04e8-a5ba-88e6-6258-d37e02990246:
       generation: 1
       last_transaction: 0
       last_xid: 0
       last_result: 0
       last_data: 0
   
     lustrefs-MDT0001-mdtlov_UUID:
       generation: 0
       last_transaction: 0
       last_xid: 0
       last_result: 0
       last_data: 0
   

2. umount (failover) the mdt target
3. force umount 1 client
4. remount the mdt target
5. wait the recovery thread to complete the recovery of the remaining client

   mdt.lustrefs-MDT0000.recovery_status=
   status: RECOVERING
   recovery_start: 86399
   time_remaining: 0
   connected_clients: 2/3
   

6. abort the mdt recovery (don't wait the missing client): OBD_INCOMPAT_MULTI_RPCS is clear by tgt_boot_epoch_update

   mds# lctl --device lustrefs-MDT0000 abort_recovery
   

    void tgt_boot_epoch_update(struct lu_target *tgt)
   ....
            if (!strncmp(tgt->lut_obd->obd_type->typ_name, LUSTRE_MDT_NAME, 3) && 
                (atomic_read(&tgt->lut_obd->obd_max_recoverable_clients) == 0 ||  
                tgt->lut_obd->obd_abort_recovery))                                
                    tgt->lut_lsd.lsd_feature_incompat &= ~OBD_INCOMPAT_MULTI_RPCS;
   

7. umount (failover) the mdt target

   last_rcvd:
     uuid: lustrefs-MDT0000_UUID
     feature_compat: 0x8
     feature_incompat: 0x21c    <-- (0x400/OBD_INCOMPAT_MULTI_RPCS is not set)
     feature_rocompat: 0x1
     last_transaction: 639950127104
     target_index: 0
     mount_count: 160
     client_area_start: 8192
     client_area_size: 128
   
     c57cde24-ed24-8705-9890-6d7bab261c25:  <-- ( with a multirpc client)
       generation: 2
       last_transaction: 0
       last_xid: 0
       last_result: 0
       last_data: 0
   
     lustrefs-MDT0001-mdtlov_UUID:
       generation: 0
       last_transaction: 0
       last_xid: 0
       last_result: 0
       last_data: 0
   

8. remount the mdt target: the lut_client_generation is reset to 0 because OBD_INCOMPAT_MULTI_RPCS is not set

   static int tgt_clients_data_init(const struct lu_env *env,
   ...
                    if (tgt->lut_lsd.lsd_feature_incompat &                     
                        OBD_INCOMPAT_MULTI_RPCS &&                              
                        lcd->lcd_generation != 0) {                             
                            /* compute the highest valid client generation */   
                            generation = max(generation, lcd->lcd_generation);  
   

9. reconnect the missing client, it should have the generation 1
10. umount/mount the client to get a duplicate generation (the generation is incremented each time):

    last_rcvd:
      uuid: lustrefs-MDT0000_UUID
      feature_compat: 0x8
      feature_incompat: 0x61c
      feature_rocompat: 0x1
      last_transaction: 644245094400
      target_index: 0
      mount_count: 161
      client_area_start: 8192
      client_area_size: 128
    
      c57cde24-ed24-8705-9890-6d7bab261c25:
        generation: 2
        last_transaction: 0
        last_xid: 0
        last_result: 0
        last_data: 0
    
      eb55f87c-d82d-5392-1016-5c7c466ceb26:
        generation: 2
        last_transaction: 0
        last_xid: 0
        last_result: 0
        last_data: 0
    
      lustrefs-MDT0001-mdtlov_UUID:
        generation: 0
        last_transaction: 0
        last_xid: 0
        last_result: 0
        last_data: 0
    

11. umount (failover) the mdt target
12. remount the mdt target --> "duplicate generation for client export"

    LustreError: 25792:0:(tgt_lastrcvd.c:1544:tgt_clients_data_init()) lustrefs-MDT0000: duplicate export for client generation 2
    LustreError: 25792:0:(obd_config.c:559:class_setup()) setup lustrefs-MDT0000 failed (-114)
    LustreError: 25792:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.0.2.4@tcp: cfg command failed: rc = -114
    Lustre:    cmd=cf003 0:lustrefs-MDT0000  1:lustrefs-MDT0000_UUID  2:0  3:lustrefs-MDT0000-mdtlov  4:f  
    

[Etienne Aujames]

This issue is likely to occur with unstable lustre client nodes or with network issues while doing failover/failback on a MDT target.

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48082
Subject: LU-15935 target: keep track of multirpc slots in last_rcvd
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2ce0684191f536b3db746648ee8a16ece5eaecab

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48400
Subject: LU-15935 target: keep track of multirpc slots in last_rcvd
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 4ccd140dc40b2f46f3508d5f1e379829aff2315b

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48082/
Subject: LU-15935 target: keep track of multirpc slots in last_rcvd
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1a79d395dd61ea2e21598bfaa5b39375e64ec22c

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49398
Subject: LU-15935 tests: add version check to replay-dual test_33
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0027fba3d3f797407fad9f3995f839a431e49782

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49399
Subject: LU-15935 target: keep track of multirpc slots in last_rcvd
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: f3bbbc6e9864509ccb18e39bb6acdd9317fa5fa5

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49400
Subject: LU-15935 tests: add version check to replay-dual test_33
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: e28bae08221829bf2dcc1201560a3598c52ad703

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49398/
Subject: LU-15935 tests: add version check to replay-dual test_33
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 92c639769d195bf3ff8c3e77c093338ac6de5e2e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49399/
Subject: LU-15935 target: keep track of multirpc slots in last_rcvd
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 756e2fc318dced505dd57e4a18bd1383f14485ce

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49400/
Subject: LU-15935 tests: add version check to replay-dual test_33
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 041c7995eb801a65f08dbbf002a282cb81f8f6a7