[LU-17240] change test-framework to format and mount targets in parallel

• Status: Open
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: WC Triage
• Resolution: Unresolved
• Labels: medium, test_script_improvements
• Severity: 3

Description

It would be useful for a number of reasons to change test-framework.sh to format and mount the MDTs and OSTs in parallel (if not both MDTs and OSTs at the same time, then at least in two sets).

• this would reduce testing time significantly for tests that reformat the filesystem (e.g. conf-sanity)
• this would improve testing of the MGS to handle registering multiple targets in parallel (there are at least some known issues with this that could be found and fixed)
• this would improve test coverage since filesystems are often mounted in parallel in production, and this would better simulate the real world

Issue Links

• is related to: LU-14928 Allow MD target re-registered after w...
• is related to: LU-4966 handle server registration errors gra...

Comments

[Patrick Farrell]

timday - I'm just living in hope here, but maybe this would be of interest to you?  It would certainly make reloading a test node faster, which would be very nice.

[Patrick Farrell]

Andreas,

Would it make sense to put "unmounting"/Stopping targets in parallel here under this same ticket?  That's closely related and also takes a while.

[Tim Day]

I've actually written some tests to do a bunch of parallel mounts, but that was client-side. It was to test out OBD device registration. I never got around to cleaning the test up and submitting it.

While mounting targets in parallel would make testing faster, I'm not sure if it would meaningfully improve test coverage. I haven't seen/heard of issues with mounting targets in parallel (even with 100s of OSS/OST). It would useful if we could find a way to register a few hundred OSS/MDS in parallel. I think that would surface more bugs faster. I think it would go:

1) Stop all clients, MDS, MDS, OSS

2) Make a bunch of small temp disks in /tmp/ on each node

3) Start a bunch of a services using those disks, hope nothing explodes

4) Cleanup and restart services

Andreas, could you link some of the known issues you mentioned (in the description) to this ticket? I'm curious what people have seen go wrong.

[Andreas Dilger]

Patrick,
yes parallel unmounting would also be useful. I think the formatting and mounting in parallel would be a bigger win.

Tim,
I don't have any tickets that have details on this, since most of the time this has happened is in conjunction with some other issue that has a higher priority to fix. Basically, what I've seen is that mounting multiple targets in parallel and registering with the MGS for the first time. If there are problems during registration (after reformat or writeconf) the MGS thinks that it is registered but the OST does not, or similar. I think shadow has previously submitted a patch for this to allow the OST to retry the initial connection, but I couldn't find it.

[Alexey Lyashkov]

Andreas,

it's Zam patch I think. I have just move all handling in single thread.
as about ticket..
I don't think it's large problem - as test target is small and format rate, except a conf-sanity.

[Andreas Dilger]

Tim, it looks like the patch I was thinking about is https://review.whamcloud.com/44594 "LU-14928 mgs: allow md target re-register" and that has been landed since 2.14.55. That at least addresses part of the issue, though I thought there was at least one more patch in this area about re-registering targets.

[Andreas Dilger]

The other ones are patch https://review.whamcloud.com/45259 "LU-15112 mgc: do not ignore target registration failure" and maybe patch https://review.whamcloud.com/45871 "LU-15112 ptlrpc: make rq_replied flag always correct" included in 2.14.57.

[Gerrit Updater]

"Timothy Day <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53518
Subject: LU-17240 tests: format and mount targets in parallel
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e2ea758f0a11cef20621759746750ac92de418af