[LU-4119] recovery time hard doesn't limit recovery duration

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.4
• Labels: patch
• Severity: 3
• Assignee: Bob Glossman (Inactive)
• Reporter: Sergey Cheremencev

Description

Firstly, I think there is a bug in extend_recovery_timer:

        if (to > obd->obd_recovery_time_hard)
                to = obd->obd_recovery_time_hard;
        if (obd->obd_recovery_timeout < to ||
            obd->obd_recovery_timeout == obd->obd_recovery_time_hard) {
                obd->obd_recovery_timeout = to;
                cfs_timer_arm(&obd->obd_recovery_timer,
                              cfs_time_shift(drt));
        }     

When "to"(recovery_timeout) will be limited by obd_recovery_time_hard, timer will be armed to (now+duration) whereas it must be armed to (recovery_start + to). I suppose following:

        if (obd->obd_recovery_timeout < to ||
            obd->obd_recovery_timeout == obd->obd_recovery_time_hard) {
                obd->obd_recovery_timeout = to;
                end = obd->obd_recovery_start + to;
                cfs_timer_arm(&obd->obd_recovery_timer, end);

But even if upper problem will be fixed, recovery will not be aborted when recovery_timeout >= time_hard.
Possible we should set obd_abort_recovery to 1 when recovery_time_hard is reached.

--- a/lustre/ldlm/ldlm_lib.c
+++ b/lustre/ldlm/ldlm_lib.c
@@ -1793,6 +1793,12 @@ static int target_recovery_overseer(struct obd_device *obd,
                                    int (*health_check)(struct obd_export *))
 {
 repeat:
+       if (cfs_time_current_sec() >=
+           (obd->obd_recovery_start + obd->obd_recovery_time_hard)) {
+               CWARN("recovery is aborted by hard timeout\n");
+               obd->obd_abort_recovery = 1;
+       }
+

Another problem is that server_cacl_timeout rewrites obd_recovery_time_hard, so we can't use proc interface to set recovery_time_hard.

Issue Links

• is duplicated by: LU-5724 IR recovery doesn't behave properly with Lustre 2.5
• is related to: LU-5986 conf-sanity test_83, test_84: failed to start OST
• is related to: LU-6084 Tests are failed due to 'recovery is aborted by hard timeout'

Comments

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13381
Subject: LU-4119 ldlm: abort recovery by time_hard
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: ade5834f5e1a6419cea47edcede01ca2f5ea79a0

[Peter Jones]

Landed for 2.7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/9078/
Subject: LU-4119 ldlm: abort recovery by time_hard
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: df89c74a320278acac7466a83393af6abd99932b