[LU-7117] replay-single test_70d: timeout and mkdir/rmdir stopped

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/56e1b56e-53ff-11e5-8f2c-5254006e85c2.

The sub-test test_70d failed with the following error:

error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d70d.replay-single/test1' (3): stripe already set
error: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed
mkdir fails
/usr/lib64/lustre/tests/replay-single.sh: line 2236: kill: (25189) - No such process
mkdir/rmdir 25189 stopped

There are several test failures and timeouts for 70d since 2015-09-02 so I suspect a patch landed on that day or the previous day that introduced a regression.

Info required for matching: replay-single 70d

Issue Links

• is duplicated by: LU-7213 replay-single test_70d: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed
• is duplicated by: LU-7924 replay-single test_70d: short description of the failure
• is related to: LU-7775 replay-single test_70d: cannot touch /mnt/lustre/d70d.replay-single/test1/a: No such file or directory
• is related to: LU-6844 replay-single test 70b failure: 'rundbench load on * failed!'
• is related to: LU-8353 mdt unlink should lock parent before lookup child
• is related to: LU-8617 replay-single test_70d: stripe already set
• is related to: LU-7172 replay-single test_70d hung on MDT unmount

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20940/
Subject: LU-7117 osp: set ptlrpc_request::rq_allow_replay properly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e3d507eec50fc1ff79acf2a9f93d52d698c887d7

[Andreas Dilger]

Never mind, I see that http://review.whamcloud.com/21064 "LU-7117 mdt: mdt unlink should lock before lookup" is abandoned since it was landed as http://review.whamcloud.com/21088 via LU-8353.

[Andreas Dilger]

How do the patches http://review.whamcloud.com/20940 "LU-7117 osp: control RPC to be sent when recovery" and http://review.whamcloud.com/21064 "LU-7117 mdt: mdt unlink should lock before lookup" relate to each other? Are they both needed? Are they different approaches to fixing the same problem, and only one is needed?

[Sebastien Buisson (Inactive)]

A lot more occurrences recently, like this one on master:
https://testing.hpdd.intel.com/test_sets/4d6f9aea-4a09-11e6-8968-5254006e85c2

[Bruno Faccini (Inactive)]

+1 on master at https://testing.hpdd.intel.com/test_sets/79710324-49a3-11e6-a80f-5254006e85c2

[Sebastien Buisson (Inactive)]

Another occurrence:
https://testing.hpdd.intel.com/test_sets/823971fa-433e-11e6-acf3-5254006e85c2

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21064
Subject: LU-7117 mdt: mdt unlink should lock before lookup
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0ae3d50ed2745366f677b79586f3fe72645330ee

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/6a54094a-3a9e-11e6-a0ce-5254006e85c2

[nasf (Inactive)]

no new RPCS (except update log redo) should be sent until the recovery is completed?

The MDT0 is in recovery, but MDT1 is normal, so the RPC from the client to MDT1 is not blocked.

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20940
Subject: LU-7117 osp: control RPC to be sent when recovery
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5fb68c49af3fa08189110a4e980ad792efd7128b

[Alex Zhuravlev]

no new RPCS (except update log redo) should be sent until the recovery is completed?