https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5662 Lustre <p>Running mkfs.lustre with the --replace flag causes a difficult-to-untangle lockout situation if the lustre MDS has never seen the index before.</p> <p>I stumbled across this while scripting the reformatting operations of a test system. In this process, I ended up reformatting a completely new OST for the first time, using the --replace flag.</p> <p>When I did this, the attempt to mount the OST on the OSS resulted in the following failure message:</p> <p>mount.lustre: mount /dev/sdae at /lustre/ost6 failed: No such file or directory<br/> Is the MGS specification correct?<br/> Is the filesystem name correct?</p> <p>Once this has happened, the MDS seems to be stuck in a halfway state, and there is no easy way recover use of the index number. If you reformat (again) with --replace specified, the error persists. If you reformat (again) with --replace absent, you get the following error on mount:</p> <p>mount.lustre: mount /dev/sdac at /lustre/ost4 failed: Address already in use<br/> The target service's index is already in use. (/dev/sdac)</p> <p>Workaround is to dismount all Lustre disks, shut down lustre with lustre_rmmod, and then remount all disks.</p> <p>This is similar to the -U versus -i flag in the rpm command. Technically, -U should not be used if the RPM has never been installed before, but the rpm command is intelligent enough to interpret -U as meaning -i if there is no prior installed version of the RPM. What rpm should certainly NOT do is to treat a technical misuse of -U on an uninstalled module as a persistent error that prevents subsequent use of either -U or -i on that module without a full reboot of Linux.</p> <p>There are numerous ways to fix this. The simplest from a user perspective is this: if a formatted OST is flagged with --replace, then when mounting, the MDS should recognize it has never seen the index before, and simply ignore the --replace tag. This would be the analogue of rpm seeing the -U flag, recognizing that it has never seen the RPM before, and simply treating -U as -i.</p> RHEL 6.5, Lustre 2.5.3 LU-5662

mkfs.lustre --replace can cause index numbers to be locked out

Bug Minor Resolved Low Priority John Fuchs-Chesney Alexander Mitin Wed, 17 Sep 2014 14:37:42 +0000 Wed, 6 Jan 2016 00:23:29 +0000 Wed, 6 Jan 2016 00:23:29 +0000 Lustre 2.5.3 0 5 <p>Joseph,</p> <p>Can you let us know if you are using IML? (Generally we'd expect IML to replace the need for this kind of scripting).</p> <p>Thanks,<br/> ~ jfc.</p> <p>Not using IML.</p> <p>Joseph,<br/> the <tt>mkfs.lustre --replace</tt> option is intended to be used in extraordinary circumstances only (i.e. if an OST has suffered fatal corruption without any chance of recovery and needs to be replaced). I agree that it should be more robust in the case you describe, but I just wanted to make sure that you are using it appropriately. </p> <p>Understood.</p> <p>I'm in a test environment where I will be breaking down and building up Lustre frequently, and after testing, anticipate needing to restart from zero: the simplest way to do this is to reformat the disks, which would normally include the MDTs as well.</p> <p>However, in stumbling through my first attempts to make this easier for myself in the future, I tripped over this bug, where using --replace in the case of a disk which has never before been formatted effectively burns the index number: you can't recover from the mistake by reformatting, because, somehow, Lustre "remembers" that you screwed up. I thought it should be documented.</p> <p>Thanks for the update. I wasn't sure if there was some use that we hadn't anticipated, so it is good to know that this is an unlikely scenario. I think this should be fixed at some point since it may be possible to hit this under normal usage also, but it doesn't sound urgent.</p> <p>No problem. And I agree, it isn't urgent.</p> <p>Is this ticket ready to be marked as resolved?</p> <p>Thanks,<br/> ~ jfc.</p> <p>I've not seen any resolution. It's still a (low priority) bug.</p> <p>Assigned back to me.<br/> ~ jfc.</p> <p>I'm marking this as resolved/low priority, since it is a rare/unanticipated case, and an unintended use of the mkfs.lustre --replace command.</p> <p>If anyone disagrees, please shout.</p> <p>~ jfc.</p> Development Rank 1|hzwwgn: Rank (Obsolete) 15764 Severity