https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6553 Lustre <p>The patch for <a href="https://jira.whamcloud.com/browse/LU-5573" title="Test timeout conf-sanity test_41c" class="issue-link" data-issue-key="LU-5573"><del>LU-5573</del></a> (<a href="http://review.whamcloud.com/#/c/12353/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12353/</a>), which closed <a href="https://jira.whamcloud.com/browse/LU-5299" title="osd_start() LBUG when doing parallel mount of the same target" class="issue-link" data-issue-key="LU-5299"><del>LU-5299</del></a>, does not cover some cases.</p> <p>Specifically, the code which enables the combined MGT/MDT to start correctly also disables the race protection for a combined MGT/MDT.</p> <p>So racing multiple mount commands on a combined MGT/MDT can still cause this problem.</p> <p>I've taken a look, and I don't see any easy way to fix this in the current context. I can provide dumps if needed, and I'll attach a log now.</p> <p>Note the attempts to start MDT0000. There are five, four of which start after the first one but before it has completed.</p> Combined MGT/MDT, racing multiple mount commands. LU-6553

Recurrence of LU-5299: obd_mount_server.c:1690:osd_start()) ASSERTION( obd ) failed

Bug Major Resolved Cannot Reproduce Bruno Faccini Patrick Farrell Fri, 1 May 2015 14:59:00 +0000 Wed, 13 Oct 2021 02:27:01 +0000 Wed, 13 Oct 2021 02:27:01 +0000 Lustre 2.7.0 Lustre 2.5.4 0 4 <p>Hello Patrick,<br/> As I am the unfortunate author of both+complementary patches for <a href="https://jira.whamcloud.com/browse/LU-5299" title="osd_start() LBUG when doing parallel mount of the same target" class="issue-link" data-issue-key="LU-5299"><del>LU-5299</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5573" title="Test timeout conf-sanity test_41c" class="issue-link" data-issue-key="LU-5573"><del>LU-5573</del></a>, I think I had to assign this ticket to me ...<br/> Thanks for the report and also the attached Lustre debug trace for the LBUG.<br/> Having a look to the trace I think you are right with the fact that the problem/race is still being present, but this is only in the case of concurrent mount/start commands where either of the nosvc/nomgs flags has been specified for a combined MDT/MGS device.<br/> Will try to fix this case too, as a new complementary patch ...</p> <p>Thanks, Bruno - Good luck. I couldn't find an easy way to do it, but I expect you know this code much better than me.</p> <p>Hi Bruno, any progress on this one? Thanks.</p> <p>Patrick, Wally,<br/> Sorry I am late on this, but back working on a new/addon patch to fully fix.<br/> Can you help in providing your reproducer or at least give some details on how these concurrent mount cmds are generated ?</p> <p>Bruno -</p> <p>We don't have a specific reproducer. It actually turned out we were doing concurrent mounts because our failover stuff was misconfigured on an internal system.<br/> But I think it would be sufficient to simply use a bash script to spawn off multiple mount commands for a particular target.</p> <p>Bruno,<br/> Here is a simple reproducer:</p> <p>1. create and start a Lustre file system with mgt/mdt combo<br/> 2. umount the mgt and mdt<br/> 3. run the following 'test_mount' script 5 times in parallel:</p> <p>cat test_mount<br/> #!/bin/bash<br/> mount -t lustre -o nosvc,abort_recov --verbose /dev/sdd /tmp/lustre/scratch/mgt<br/> mount -t lustre -o nomgs,abort_recov --verbose /dev/sdd /tmp/lustre/scratch/mdt</p> <p>for ((i=0;i&lt;5;i++));do ./test_mount &amp; done;</p> <p>Thanks, Wally!</p> Related LU-5573 LU-5299 Development Rank 1|hzxcc7: Rank (Obsolete) 9223372036854775807 Severity