https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17240 Lustre <p>It would be useful for a number of reasons to change <tt>test-framework.sh</tt> to format and mount the MDTs and OSTs in parallel (if not both MDTs and OSTs at the same time, then at least in two sets).</p> <ul class="alternate" type="square"> <li>this would reduce testing time significantly for tests that reformat the filesystem (e.g. conf-sanity)</li> <li>this would improve testing of the MGS to handle registering multiple targets in parallel (there are at least some known issues with this that could be found and fixed)</li> <li>this would improve test coverage since filesystems are often mounted in parallel in production, and this would better simulate the real world</li> </ul> LU-17240

change test-framework to format and mount targets in parallel

Improvement Minor Open Unresolved WC Triage Andreas Dilger medium test_script_improvements Sat, 28 Oct 2023 10:26:35 +0000 Wed, 20 Dec 2023 19:42:05 +0000 0 6 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=timday" class="user-hover" rel="timday">timday</a> - I'm just living in hope here, but maybe this would be of interest to you?  It would certainly make reloading a test node faster, which would be very nice.</p> <p>Andreas,</p> <p>Would it make sense to put "unmounting"/Stopping targets in parallel here under this same ticket?  That's closely related and also takes a while.</p> <p>I've actually written some tests to do a bunch of parallel mounts, but that was client-side. It was to test out OBD device registration. I never got around to cleaning the test up and submitting it.</p> <p>While mounting targets in parallel would make testing faster, I'm not sure if it would meaningfully improve test coverage. I haven't seen/heard of issues with mounting targets in parallel (even with 100s of OSS/OST). It would useful if we could find a way to register a few hundred OSS/MDS in parallel. I think that would surface more bugs faster. I think it would go:</p> <p>1) Stop all clients, MDS, MDS, OSS</p> <p>2) Make a bunch of small temp disks in /tmp/ on each node</p> <p>3) Start a bunch of a services using those disks, hope nothing explodes</p> <p>4) Cleanup and restart services</p> <p>Andreas, could you link some of the known issues you mentioned (in the description) to this ticket? I'm curious what people have seen go wrong.</p> <p>Patrick,<br/> yes parallel unmounting would also be useful. I think the formatting and mounting in parallel would be a bigger win.</p> <p>Tim,<br/> I don't have any tickets that have details on this, since most of the time this has happened is in conjunction with some other issue that has a higher priority to fix. Basically, what I've seen is that mounting multiple targets in parallel and registering with the MGS for the first time. If there are problems during registration (after reformat or writeconf) the MGS thinks that it is registered but the OST does not, or similar. I think <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=shadow" class="user-hover" rel="shadow">shadow</a> has previously submitted a patch for this to allow the OST to retry the initial connection, but I couldn't find it. </p> <p>Andreas,</p> <p>it's Zam patch I think. I have just move all handling in single thread.<br/> as about ticket..<br/> I don't think it's large problem - as test target is small and format rate, except a conf-sanity.</p> <p>Tim, it looks like the patch I was thinking about is <a href="https://review.whamcloud.com/44594" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44594</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14928" title="Allow MD target re-registered after writeconf" class="issue-link" data-issue-key="LU-14928"><del>LU-14928</del></a> mgs: allow md target re-register</tt>" and that has been landed since 2.14.55. That at least addresses part of the issue, though I thought there was at least one more patch in this area about re-registering targets.</p> <p>The other ones are patch <a href="https://review.whamcloud.com/45259" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45259</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15112" title="attempt to register an OST with duplicated index should fail but it does not" class="issue-link" data-issue-key="LU-15112"><del>LU-15112</del></a> mgc: do not ignore target registration failure</tt>" and maybe patch <a href="https://review.whamcloud.com/45871" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45871</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15112" title="attempt to register an OST with duplicated index should fail but it does not" class="issue-link" data-issue-key="LU-15112"><del>LU-15112</del></a> ptlrpc: make rq_replied flag always correct</tt>" included in 2.14.57.</p> <p>"Timothy Day &lt;timday@amazon.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53518" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53518</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17240" title="change test-framework to format and mount targets in parallel " class="issue-link" data-issue-key="LU-17240">LU-17240</a> tests: format and mount targets in parallel<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e2ea758f0a11cef20621759746750ac92de418af</p> Related LU-14928 LU-4966 Development Rank 1|i03zwn: Rank (Obsolete) 9223372036854775807 Severity