https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8953 Lustre <p>The MDT for one of our filesystems is full, and it's not possible to delete any files, rendering the filesystem unusable from the users point of view.</p> <p>It's possible to manually track files that could be deleted via fid to ZFS objects on the disk. But we haven't found a way to delete objects via zdb. A recovery procedure using something like that would probably be good to have if more people run in to this. </p> <p>Given that it's almost Christmas vacation, so lets keep this simple and low risk. I've thrown some more disks into the MDS. Given that the filesystem with problems looks like this:</p> <p> lustre-mdt0 ONLINE 0 0 0<br/> mirror-0 ONLINE 0 0 0<br/> mds9_sdm-mdt_fouo6_sdm ONLINE 0 0 0<br/> mds9_sdn-mdt_fouo6_sdn ONLINE 0 0 0<br/> mirror-1 ONLINE 0 0 0<br/> mds9_sdo-mdt_fouo6_sdo ONLINE 0 0 0<br/> mds9_sdp-mdt_fouo6_sdp ONLINE 0 0 0</p> <p>Would it be safe (and fix the problem) to expand it by adding another mirror?:</p> <p>zpool add lustre-mdt0 mirror /dev/exp_sdq/mdt_fouo6exp_sdq /dev/exp_sdt/mdt_fouo6exp_sdt</p> <p>(This is probably the same issue as <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a>, so feel free to merge them if it makes sense.)</p> Centos 6, Lustre from llnl chaos branch LU-8953

ZFS-MDT 100% full. Request for verification of plan to fix

Bug Blocker Resolved Done Nathaniel Clark Peter Bortas Mon, 19 Dec 2016 15:13:47 +0000 Tue, 20 Dec 2016 15:37:50 +0000 Tue, 20 Dec 2016 15:37:50 +0000 Lustre 2.5.3 0 4 <p>Addendum: Our intention is to expand the pool without shutting down Lustre. Either way <em>should</em> be fine, but expanding it live and giving Lustre at least the chance of completing any outstanding operations feels like the more sound way. Please let us know if you disagree.</p> <p>Nathaniel</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a>,</p> <p>I know taking the FS down and growing the MDT will alleviate your issue. I _think growing the MDT live will be okay, but I would want to double check (run a test locally) before I could bless that course of action.</p> <p>&#8211;<br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a></p> <p>Nathaniel,</p> <p>When do you think you could run that test?</p> <p>After having talked through the failure scenarios of shutting down the FS in this state we decided to do it after unmounting since it's the procedure you know works. Seems to have worked out without any failures I can detect so far. For the record this is what we did:</p> <p>umount lustre-mdt0/fouo6<br/> zpool add lustre-mdt0 mirror /dev/exp_sdq/mdt_fouo6exp_sdq /dev/exp_sdt/mdt_fouo6exp_sdt<br/> mount -t lustre lustre-mdt0/fouo6 /mnt/lustre/local/fouo6</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a>,</p> <p>I'm glad that worked for you. I'll close this bug seeing as you've completed your expansion of the MDT. If I'm mistaken that you need something else from this bug, please feel free to re-open.</p> <p>&#8211;<br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a></p> Development Rank 1|hzyypj: Rank (Obsolete) 9223372036854775807 Severity