https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9023 Lustre <p>This is a sanity check question. NSC sees no reason the method described below should not work, but due to the high impact a failure would have we'd like a second opinion. We have scheduled downtime to execute it Thursday next week, 26 Jan.</p> <p>To sort out the fallout of <a href="https://jira.whamcloud.com/browse/LU-8953" title="ZFS-MDT 100% full. Request for verification of plan to fix" class="issue-link" data-issue-key="LU-8953"><del>LU-8953</del></a> (out of inodes on ZFS MDT solved by adding more disks to the pool) we need to recreate the original pool. The reason we ran out of inodes was that when the vendor sent us hardware for the latest expansion that was supposed to be equivalent to the last shipment the SSD had switched from reporting 512b blocks to 4k blocks. Since I had not hardcoded ashift we ended up with 6-8 times fewer inodes and this was missed in testing.</p> <p>There isn't enough slots in the MDSs to solve this by throwing HW at it as a permanent solution, so I need to move all data from pools with ashift=12 to ashift=9. Do you see any problem with just doing the following:</p> <p>(The funny device names come from running LVM just to get more easily identifiable names)</p> <p>Unmount the filesystem on all nodes then run something like this for each mdt that needs fixing:</p> <p>umount lustre-mdt0/fouo6<br/> zfs snapshot lustre-mdt0/fouo6@copythis<br/> zpool create lustre-mdt-tmp -o ashift=9 mirror \<br/> /dev/new_sdr/mdt_fouo6new_sdr \<br/> /dev/new_sdu/mdt_fouo6new_sdu<br/> zfs send -R lustre-mdt0/fouo6@copythis | zfs recv lustre-mdt-tmp/fouo6tmp<br/> zpool destroy lustre-REMOVETHIS-mdt0<br/> zpool create lustre-mdt0/fouo6 -o ashift=9 \<br/> mirror /dev/mds9_sdm/mdt_fouo6_sdm /dev/mds9_sdn/mdt_fouo6_sdn \<br/> mirror /dev/mds9_sdo/mdt_fouo6_sdo /dev/mds9_sdp/mdt_fouo6_sdp<br/> zfs send -R lustre-mdt-tmp/fouo6tmp@copythis | zfs recv lustre-mdt0/fouo6<br/> mount -t lustre lustre-mdt0/fouo6 /mnt/lustre/local/fouo6<br/> zpool destroy lustre-mdt-tmp</p> <p>The "REMOVETHIS-" inserted due to desktop copy buffer paranoia should be removed before running of course.</p> LU-9023

Second opinion on MDT inode recovery requested

Question/Request Major Open Unresolved Gabriele Paciucci Peter Bortas Mon, 16 Jan 2017 14:56:37 +0000 Tue, 31 Jan 2017 13:50:56 +0000 0 6 <p>That create line is incorrect. Should be just "zpool create lustre-mdt0" without the extra filesystem part.</p> <p>Hi Zhiqi,</p> <p>Do you have any recommendation here?</p> <p>Thanks.<br/> Joe</p> <p>Peter,<br/> While we wait for Zhiqi to comment, you can also see the commentary in <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> as a 2nd source of information.<br/> Thanks.<br/> Joe</p> <p>Just a test to check access for <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a></p> <p>Appriciated Joseph,</p> <p>That doc in my mind confirms that we are on the right track with this procedure. I'll wait for Zhiqi to see if he has any further insight. </p> <p>(And thanks Peter, I can see the tickets again now.)</p> <p>Cheers,<br/> Peter B</p> <p>Hi Peter B, </p> <p>Gabriele will try this procedure in an internal develop lab and update this ticket with his experience. We understand your timing, "We have scheduled downtime to execute it Thursday next week, 26 Jan."</p> <p>We should have results in one day or two. </p> <p>Best Regards,<br/> Zhiqi</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a>,<br/> I would say maybe 3 or 4 days <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>BTW I'm London based, so we can organize a call to double check the procedure.</p> <p>Hi Gabriele,</p> <p>Sounds great. Let's check in on Monday and see how things have progressed. And we can schedule a call then if we feel it's needed.</p> <p>Okay this is a first procedure that doesn't need a second pool and we save our backup in a gzip file. In my environment, I have in the same pool MDT and MGT.<br/> The name of the pool is <tt>MDS</tt></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zfs snap -r MDS@backup # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT MDS@backup 0 - 96K - MDS/mdt0@backup 0 - 489M - MDS/mgt@backup 0 - 5.27M - </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zfs send -R MDS@backup | gzip &gt; /tmp/backup.gz # zfs list NAME USED AVAIL REFER MOUNTPOINT MDS 495M 360G 96K /MDS MDS/mdt0 489M 360G 489M /MDS/mdt0 MDS/mgt 5.27M 360G 5.27M /MDS/mgt </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zpool destroy MDS # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zpool create -o ashift=9 MDS mirror /dev/sdc /dev/sde # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT MDS 372G 50K 372G - 0% 0% 1.00x ONLINE - </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zcat /tmp/backup.gz | zfs recv -F MDS # zfs list NAME USED AVAIL REFER MOUNTPOINT MDS 121M 360G 19K /MDS MDS/mdt0 118M 360G 118M /MDS/mdt0 MDS/mgt 3.22M 360G 3.22M /MDS/mgt </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT MDS@backup 0 - 19K - MDS/mdt0@backup 0 - 118M - MDS/mgt@backup 0 - 3.22M - </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre MDS/mgt /mnt/mgt/ # mount -t lustre MDS/mdt0 /mnt/mdt0 # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda4 897134592 3685648 893448944 1% / devtmpfs 32823216 0 32823216 0% /dev tmpfs 32836956 39648 32797308 1% /dev/shm tmpfs 32836956 9444 32827512 1% /run tmpfs 32836956 0 32836956 0% /sys/fs/cgroup /dev/sda2 10471424 176380 10295044 2% /boot /dev/sda1 1046516 9644 1036872 1% /boot/efi tmpfs 6567392 0 6567392 0% /run/user/0 MDS/mgt 374806016 3328 374800640 1% /mnt/mgt MDS/mdt0 374922752 120960 374799744 1% /mnt/mdt0 </pre> </div></div> <p>Do you need the same procedure using another zpool ?</p> <p>Not really. I like your method better. It does invalidate some of my testing though, so I'll run some new over the weekend.</p> <p>Okay, I'm now on hold waiting for your feedback. </p> <p>Hi Gabriel,</p> <p>The weekends tests looks good. I have some tests I will run over night and lock down the plans tomorrow. A couple of questions:</p> <p>1. Did you have any reason that sending the whole pool would be better than sending individual filesystems except that it was easier because you also had the MGT there? Unless there is a reason not to I will send the filesystems, only for clearitys sake. The pools have anonymous names while the MDTs are named after the filsystems. I will be doing this for 3 pools on the same machine, so keeping the names reduces the chance of recv:ing or destroying the wrong filesystem. These will be the actual sends on my end:</p> <p> zfs send -vR lustre-mdt0/fouo6@copythis | gzip &gt; /lustre-mdt-tmpfs/mds0-fouo6.gz <br/> zfs send -vR lustre-mdt1/rossby20@copythis | gzip &gt; /lustre-mdt-tmpfs/mds1-rossby20.gz <br/> zfs send -vR lustre-mdt2/smhid13@copythis | gzip &gt; /lustre-mdt-tmpfs/mds2-smhid13.gz </p> <p>2. I will not be moving the MGT from ashift=12 to ashift=9. Will this cause any problems? I know the question is borderline insane, but this is really the original reason I opened this ticket with you. I'm OK with sorting out everything on the zfs level, but I'm trying to fish for half-insane things like hard-coding offsets on MDT creation time based on number of blocks somewhere deep in Lustre.</p> <p>Peter</p> <p>Gabriele is unexpectedly out of the office at short notice. Can this wait until he is available again (hopefully next week)?</p> <p>Peter</p> <p>Hi Peter,</p> <p>That's unfortunate. Of course it's technically possible to delay this to another week, but the cluster downtime is now to late to stop for this week. I will also have to mount the filesystems ro for a few weeks since the users will run out of inodes before the next window.</p> <p>I'd be happy with an answer to just this question: As far as Intel engineers know, is there anything in the filesystem that stores a structure that would be affected by a change in block size; i.e. could cause problems during this data move. We'll assume for the sake of this discussion that I'll be able to flawlessly take care of the bit shuffling on disk.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a>,<br/> I'm back...sorry for this. I don't know if this is too late:<br/> 1. Yes I was sending the whole pool for that reason, but testing only the individual volume worked. But I suggest to have a backup of the whole file system is not a bad idea...just in case<br/> 2. We are not expecting any performance or big capacity requirement on the MGT, so I don't see any problem to leave with the original ashift.</p> <p>Making lustre decide the shift layout at the format time it is something that maybe <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> can evaluate. Not sure if Lustre can evaluate the physical layout of the disks.</p> <p>Hi Gabriel,</p> <p>You are in time. We got a bit delayed by hardware failing elsewhere in the cluster, so the procedure is just started. We'll know today if I lost the filesystems or not.</p> <p>I'll make an extra backup of the whole filesystems. It just adds about 1h to the procedure, and that's worth it.</p> <p>I don't think the tools for formatting really needs any intelligence here really, this was an operator error. But if there are no performance problems with running ashift=9 on 4k block SSDs in the general case it might be a good idea to default to ashift=9 there though. In my tests I've not seen any performance advantage outside of the error margin by using ashift=12 on SSDs on the MDS.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zino" class="user-hover" rel="zino">zino</a>,<br/> you don't see performance improvement because the bottleneck is in the code and not in the underline hw performance. We saw very different performance instead in the OST not using ashift=12.</p> <p>This operation was somewhat delayed by unrelated failures in one of the attached compute clusters, but completed without problems on Friday.</p> <p>I have noted one oddity with ZFS snapshots today, but nothing that affects production. I'll try to figure out that one by tomorrow and then we can close this.</p> <p>The ZFS oddity seems to be unrelated to the recreation of the filesystems, so I'll track that separately if needed.</p> <p>This concludes this issue from my side. Thanks for the help everyone!</p> Related LUDOC-161 Development Rank 1|hzz0sf: Rank (Obsolete) 9223372036854775807