[LU-15934] client refused mount with -EAGAIN because of missing MDT-MDT llog connection

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Andreas Dilger

Description

New clients were unable to establish a connection to the MDT, even after recovery had been aborted due to an llog context not being set up properly. The clients were permanently getting -11 = -EAGAIN errors from the server:

(service.c:2298:ptlrpc_server_handle_request()) Handling RPC req@ffff8cdd37ad0d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt09_0
01:0+-99:4093:x1719493089340032:12345-10.16.172.159@tcp:38:
(service.c:2303:ptlrpc_server_handle_request()) got req 1719493089340032
(tgt_handler.c:736:tgt_request_handle()) Process entered
(ldlm_lib.c:1100:target_handle_connect()) Process entered
(ldlm_lib.c:1360:target_handle_connect()) lfs02-MDT0003: connection from 16778a5c-5128-4231-8b45-426adc7e94b6@10.16.172.159@tcp t55835524055 exp           (null) cur 51537 last 0
(obd_class.h:831:obd_connect()) Process entered
(mdt_handler.c:6671:mdt_obd_connect()) Process entered
(lod_dev.c:2136:lod_obd_get_info()) lfs02-MDT0003-mdtlov: lfs02-MDT0001-osp-MDT0003 is not ready.
(lod_dev.c:2145:lod_obd_get_info()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5)
(ldlm_lib.c:1446:target_handle_connect()) Process leaving via out (rc=18446744073709551605 : -11 : 0xfffffffffffffff5)
(service.c:2347:ptlrpc_server_handle_request()) Handled RPC req@ffff8cdd37ad0d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt09_001:0+-99:4093:x1719493089340032:12345-10.16.172.159@tcp:38: Request processed in 86us (124us total) trans 0 rc -11/-11

This corresponds to the following block of code in lod_obd_get_info(), where it is the second "is not ready" message being printed from the missing ctxt->loc_handle:

                lod_foreach_mdt(d, tgt) {
                        struct llog_ctxt *ctxt;
        
                        if (!tgt->ltd_active)
                                continue;
               
                        ctxt = llog_get_context(tgt->ltd_tgt->dd_lu_dev.ld_obd,
                                                LLOG_UPDATELOG_ORIG_CTXT);
                        if (!ctxt) {
                                CDEBUG(D_INFO, "%s: %s is not ready.\n",
                                       obd->obd_name,
                                      tgt->ltd_tgt->dd_lu_dev.ld_obd->obd_name);
                                rc = -EAGAIN;
                                break;
                        }
                        if (!ctxt->loc_handle) {
                                CDEBUG(D_INFO, "%s: %s is not ready.\n",
                                       obd->obd_name,
                                      tgt->ltd_tgt->dd_lu_dev.ld_obd->obd_name);
                                rc = -EAGAIN;
                                llog_ctxt_put(ctxt);
                                break;
                        }
                        llog_ctxt_put(ctxt);
                }

It would be useful to distinguish those two messages more clearly, e.g. "ctxt is not ready" and "handle is not ready", as minor differences in line numbers would make it difficult to distinguish them in the logs.

The root problem is that the MDT0003-MDT0001 connection wasn't completely set up due to abort_recovery_mdt (due to a different recovery error, LU-15761), and the MDS never retries to establish this connection, leaving the filesystem permanently unusable. Running "lctl --device NN recover" reconnected the import, but did not actually re-establish the llog context. Mounting with "-o abort_recov_mdt" resulted in the problem moving to MDT0000 (only the first bad llog context is printed before breaking out of the loop).

I think there are two issues to be addressed here:
1) the MDS should try to reconnect and rebuild the llog connection in this case, at least on recover if not automatically. there didn't appear to be any permanent reason why these llog connections were not working, just fallout from abort_recovery_mdt.
2) is it strictly necessary to block client mounting if not all MDT-MDT connections are established? Or is no different than any other case where the MDT loses a connection after it is mounted? The MDT recovery had already been aborted, so allowing new clients to connect shouldn't cause any issues. Maybe this issue would be moot if (1) was fixed, but it seems otherwise counter productive. The filesystem was apparently fully functional for clients that had previously mounted before the MDT recovery (both MDT0003/MDT0001 and MDT0001/MDT0003 remote directory creation worked fine).

Issue Links

• is related to: LU-15938 MDT recovery did not finish due to corrupt llog record
• is related to: LU-16159 remove update llog files after recovery abort
• is related to: LU-17365 steady LOD update llog connection

Comments

[Andreas Dilger]

YS, can you see why mds0 was not finished recovery? If it was making progress, then waiting longer would be OK (VM testing can be very unpredictable). However, if it is stuck for some other reason then waiting will not help and the blocker to finish recovery needs to be fixed.

[Yang Sheng]

Hi, Andreas,

Looks like the mds0 was still waiting for recovery. But mds1 was not blocked on lod part rather than communication. Do we need prolong the waiting time?

Thanks,
YangSheng

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51208/
Subject: LU-15934 tests: add a test case for update llog
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 54301fe4f598eef5aebdbdb0c7f3dddea9541c4e

[Xing Huang]

2023-06-25: The second patch (#49528) landed to master, the third patch(#51208) adding test case is ready to land(on master-next branch)

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49528/
Subject: LU-15934 lod: clear up the message
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9882d4e933fd8cdbc4a9bc8bf6b29655009f7e03

[Xing Huang]

2023-06-17: The second patch (#49528) is ready to land(on master-next branch), the third patch adding test case is being reviewed

[Gerrit Updater]

"Yang Sheng <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51208
Subject: LU-15934 tests: add a test case for update llog
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 48ec3ead32ff44f33997051752601009903684e9

[Xing Huang]

2023-05-20: The improving patch(#49569) landed to master, another patch(#49528) is being discussed.

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49569/
Subject: LU-15934 lod: renew the update llog
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 814691bcffab0a19240121740fb85a1912886a3c