https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10946 Lustre <p>During our benchmarking/testing, we found sometimes write performances are not stable enough and there are some small read during write which could drop thoughoutput of write performances.</p> <p>It turned out that block bitmaps load make some latency here, also for a heavy fragment filesystem, we might need load many bitmaps to find some free blocks.</p> <p>To improve above situation, we had a patch to load block bitmaps to memory and pin those bitmaps memory until unmount or we release the memory on purpose, this could stable write performances and improve performances of a heavy fragment filesystem.</p> LU-10946

add an interface to load ldiskfs block bitmaps

Improvement Minor Closed Won't Fix Wang Shilong Wang Shilong patch Tue, 24 Apr 2018 14:32:27 +0000 Thu, 28 Sep 2023 02:53:15 +0000 Mon, 20 Apr 2020 01:34:32 +0000 0 12 <p>I need cleanup our internal patch a bit, will push the patch to master very soon.</p> <p>ok - thanks <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=wangshilong" class="user-hover" rel="wangshilong">wangshilong</a></p> <p> </p> <p>This feature is of interest to NASA. (We currently use scripts around debugfs run periodically from cron to dump the bitmap information, and also the object trees with 'debugfs -c -R "ls O/0/d$i"'.)</p> <p>I found this was discussed way back in <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a>, with memory requirement estimates (are they still valid?):<br/> <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a> <a href="#comment-12883" target="_blank" rel="noopener">comment-12883</a><br/> Also from ticket <a href="https://jira.whamcloud.com/browse/LU-3631" title="file stats are slow on filesystem start up" class="issue-link" data-issue-key="LU-3631"><del>LU-3631</del></a> is reading the inode bitmaps no longer really useful?</p> <p>Regarding this patch, I think it would be helpful to include options to make it more generally configurable and usable for a wider variety of use-cases, not just pin block bitmaps. For example...</p> <ul> <li>Just do a pre-read of the bitmaps to warm the cache, without pinning.</li> <li>An option to either load at mount time, or load on demand.</li> <li>Differentiate between loading (and pinning) Data Block bitmap vs. Inode bitmap vs. the full Inode Table.</li> </ul> <p>Thanks!</p> <p>Hi Nathan Dauchy,</p> <p> Thanks for your input here, looks you raised a lot of interesting question here. ^_</p> <p>1)regarding memory requirement, yup, there will be memory pressure for pinning bitmaps.<br/> just considering we have one block bitmap(4K) vs a 128M block group, it might eat a lot<br/> of memory if system is big. and same requirement for inode bitmap. I guess system memory<br/> might not be enough to pin both inode bitmap and block bitmaps.</p> <p>2)we tried to just read bitmaps to memory and make it reclaimable but write performances<br/> still not stable since those bitmap memory reclaimed easier before we want it.</p> <p>3)I don't think it a good idea to load(pin) full inode table, since full inode table eat much more memory<br/> than bitmaps..maybe some on-demand load inode table ahead makes sense, which I am not sure.</p> <p>So I agreed we can make the patch more configurable with:<br/> 1)pre-read of bitmap to warm the cache without pinning.<br/> 2)option to pin bitmap</p> <p>Thanks,<br/> Shilong</p> <p>Discussions of this at LUG brought up another issue, and possible solution... for cases of failover pairs, a server may have enough memory for pinning bitmaps in normal operation but OOM in a failover event with 2X the OST count.  To handle that case, and protect low-memory OSS in general, there could also be a configurable amount (80% by default?) of total memory threshold above which not to pin bitmaps.  I don't know if it would be better to pin as much as possible up to that threshold, or to pre-calculate whether all bitmaps for a given OST are pinnable based on total (or free) memory; but either way it should report a kernel error message if not able to pin without going over.</p> <p>For clarification on the metadata pre-loading done at NASA, I have uploaded the scripts.  They are run from cron like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">0 1 * * * root /usr/local/bin/read-meta.sh dump &gt;&gt; /<span class="code-keyword">var</span>/log/lustre-read-meta.log 2&gt;&amp;1 */15 * * * * root /usr/local/bin/read-meta.sh read &gt;&gt; /<span class="code-keyword">var</span>/log/lustre-read-meta.log 2&gt;&amp;1 </pre> </div></div> <p>The original developer has left, but it sounds like the scripts were actually created with a focus on caching inode information (such as to speed up "ls -l"), not necessarily free blocks. Perhaps they have the nice side effect of refreshing the block bitmaps in cache, and if other changes in the last few years like flex_bg have improved inode table reading, then these scripts are now unnecessary?</p> <p>Shilong, any progress on this patch?</p> <p>Nathan Dauchy,</p> <p> Attachment read-inode is bad format to read, so we might better figure out caching which kind of metadata exactly improved your performances?</p> <p>It is just a text file, perl script.  The file simply uses line feed characters, but perhaps your editor (when no extension is present) is looking for carriage return too.</p> <p>Nathan,</p> <p>I mean when I downloaded the attachment it is hard for me to read it, it is something like following<br/> is a bit messy code..</p> <p>#!/usr/bin/perl -w # $Header: /cvsroot/lustre-tools/src/read-inodes,v 1.2 2011/03/16 00:04:17 jrappley Exp $ use strict; use File::Basename; use Getopt::Long; use Pod::Usage; use POSIX qw(ceil); use Fcntl 'SEEK_SET'; my $progname = basename $0; # Globals my $group = 0; my $inodesPerGroup = 0; my $inodeBlocksPerGroup = 0; my @usedInodes; my $freeIBlocks = 0; # Command line options my %arg; $arg</p> {verbose} = 0; sub dprint { if ($arg{verbose} <p> &gt; 1) </p> { print @_; } <p> } # Parse command line options Getopt::Long::Configure("bundling"); GetOptions( "h|help" =&gt; \$arg</p> {h}, "d|device=s" =&gt; \$arg{device}, "m|meta=s" =&gt; \$arg{meta}, "v|verbose+" =&gt; \$arg{verbose}, ) or pod2usage(-exitval =&gt; 2, -verbose =&gt; 1); pod2usage(1) if ($arg{h} <p>); pod2usage(1) if (scalar(@ARGV) <span class="error">Unable to render embedded object: File (= 0); pod2usage(1) if () not found.</span> defined $arg</p> {meta}); if (defined $arg{device}) { open D, "&lt;", "$arg{device}" or die "Couldn't open $arg{device}: $!"; } open M, "&lt;", $arg{meta} <p> or die "Couldn't open $arg</p> {meta} <p>: $!"; while () { if (/Inodes per group:\s+(\d+)/) </p> { $inodesPerGroup = $1; } <p> elsif (/Inode blocks per group:\s+(\d+)/) </p> { $inodeBlocksPerGroup = $1; } <p> elsif (/^$/) </p> { last; } <p> } if ($inodesPerGroup == 0 || $inodeBlocksPerGroup == 0) </p> { print STDERR "Couldn't determine number of inodes per group, exiting\n"; exit 1; } <p> my $inodesPerBlock = $inodesPerGroup / $inodeBlocksPerGroup; my $firstItableBlock; while () { if (/^Group (\d+)/) </p> { $group = $1; } <p> if (/Inode table at (\d+)/) </p> { $firstItableBlock = $1; } <p> if (/(\d+) free inodes/) </p> { $usedInodes[$group] = $inodesPerGroup - $1; } <p> if (/Free inodes:\s(.*)/) { # next if $group &gt; 1; my @usedIBlocks; my $groupFreeIBlocks = 0; for my $i (0..$inodeBlocksPerGroup - 1) </p> { $usedIBlocks[$i] = $inodesPerBlock; } <p> dprint "$group: ", join(" ", @usedIBlocks), "\n"; my @irange = split(/, /, $1); dprint "Group $group: ", join(" X ", @irange), "\n"; foreach my $range (@irange) { dprint "range: $range\n"; if ($range =~ /(\d+)-(\d+)/) { my $low = $1 - ($group * $inodesPerGroup); my $high = $2 - ($group * $inodesPerGroup); dprint "marking $low..$high\n"; for my $inum ($low..$high) </p> { my $iBlock = ceil($inum / $inodesPerBlock) - 1; dprint "inum $inum block $iBlock\n"; $usedIBlocks[$iBlock]--; } <p> } else </p> { my $inum = $range - ($group * $inodesPerGroup); my $iBlock = ceil($inum / $inodesPerBlock) - 1; dprint "marking $inum block $iBlock\n"; $usedIBlocks[$iBlock]--; } <p> } for my $i (0..$#usedIBlocks) { if ($usedIBlocks<span class="error">&#91;$i&#93;</span> == 0) { $groupFreeIBlocks++; my $block = $firstItableBlock + $i; if (0) </p> { dprint "group $group read ", $firstItableBlock + $i, "\n"; sysseek(D, $block * 4096, SEEK_SET); sysread(D, my $foo, 4096); } <p> } elsif (defined $arg</p> {device} <p>) </p> { my $block = $firstItableBlock + $i; dprint "group $group read ", $firstItableBlock + $i, "\n"; sysseek(D, $block * 4096, SEEK_SET); sysread(D, my $foo, 4096); } <p> } $freeIBlocks += $groupFreeIBlocks; my @blockStr = map </p> { sprintf "%3d", $_ } <p> @usedIBlocks; if ($arg</p> {verbose} <p>) </p> { printf("%6d: %3d/%3d | %s\n", $group, $usedInodes[$group], $groupFreeIBlocks, join("", @blockStr)); } <p> } } print "Unused inode blocks: $freeIBlocks\n"; close M; _<em>END</em>_ =head1 NAME =head1 SYNOPSIS skeleton.pl <span class="error">&#91;-h&#93;</span> =head1 DESCRIPTION =head1 OPTIONS =over 8 =item B&lt;<del>h|</del>-help&gt; Print a help message and exit. =back 8 =head1 EXAMPLES =head1 ENVIRONMENT =over 8 =item B FOO is an environment variable that somehow alters the execution of this program. =back 8 =head1 KNOWN BUGS =head1 CAVEATS =head1 DETAILS =head1 REPORTING BUGS =head1 AUTHOR =head1 SEE ALSO</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/32347" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32347</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10946" title="add an interface to load ldiskfs block bitmaps" class="issue-link" data-issue-key="LU-10946"><del>LU-10946</del></a> ldiskfs: add an interface to load ldiskfs block bitmaps<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f2989ee1ac1b7ca5666fc1cf42f9e95f3da20200</p> <p>Hi Nathan Dauchy,</p> <p> Sorry for late patch, I just pushed a simple version, could you help test if it help your performance case?</p> <p>echo 1 &gt; /sys/fs/ldiskfs/vdb/loadbbitmaps # this is just load not pin.<br/> echo 2 &gt; /sys/fs/ldiskfs/vdb/loadbbitmaps # pin block bitmaps in memory<br/> echo 0 &gt; /sys/fs/ldiskfs/vdb/loadbbitmaps #unpin block bitmaps in memory</p> <p>Thanks,<br/> Shilong</p> <p>Hi Shilong,</p> <p>Nathan asked me to cherry-pick the #32347 review. This patch caused conflicts in b2_10. Branch b2_10 is 4 ldiskfs kernel_patches behind compared to master branch. Is there dependency on the 4 missing patches or any other commit? If yes, could you list prerequisites of your patch?</p> <p>Thanks,<br/> Jay</p> <p>Hi Jay Lan,</p> <p> You could just ignore the 4 missing patches, and apply my patch directly, I build locally, it works.</p> <p>Thanks,<br/> Shilong</p> Related LU-17153 LU-10967 LU-12970 Development Rank 1|hzzw9z: Rank (Obsolete) 9223372036854775807