https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2049 Lustre <p>Currently, grant is still inflated if backend block size &gt; page size (that's the case with zfs osd).<br/> OBD_CONNECT_GRANT_PARAM was added to address this and we need to develop the the code in osc &amp; ofd to implement support for this feature.</p> LU-2049

add support for OBD_CONNECT_GRANT_PARAM

Technical task LU-3522 Critical Resolved Fixed Nathaniel Clark Johann Lombardi HB llnl prz Fri, 28 Sep 2012 17:10:48 +0000 Fri, 15 Oct 2021 18:06:08 +0000 Wed, 24 May 2017 12:20:13 +0000 Lustre 2.4.0 Lustre 2.5.0 Lustre 2.10.0 0 23 <p>To further clarify - the whole reason that there is grant inflation with ZFS is because clients currently only consume grant in PAGE_SIZE chunks (i.e. typically 4kB units), since there are no native Linux filesystems with blocksize &gt; PAGE_SIZE. This problem could also be hit with IA64/PPC/SPARC servers with PAGE_SIZE = 64kB and ext4 having data blocks this large, or with ext4's "bigalloc" feature, so this is not necessarily a ZFS-only bug, just that ZFS is the first OSD used with larger blocksize.</p> <p>Having server blocksize &gt; client PAGE_SIZE means (in the worst case) if some client is writing sparse PAGE_SIZE chunks into files, the client's smaller write might consume a full block of space on the OST. Without the server-side grant inflation this could lead to the OST incorrectly running out of space before the client runs out of grant, and lose writeback cached data on the client.</p> <p>In order to fix this problem, client RPCs need to be modified to consume grant in units as given by ocd_inodespace, ocd_blocksize, and ocd_grant_extent when OBD_CONNECT_GRANT_PARAM is set. For each object modified by a write, ocd_inodespace is consumed. For data, the minimum chunk is sized and aligned on ocd_blocksize. Additionally, for each discontiguous extent of data (including the first one) consume ocd_grant_extent worth of space.</p> <p>I tried printing out the current values assigned to these fields for a ZFS filesystem using <a href="http://review.whamcloud.com/6588" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6588</a>, but this showed all of these fields as zero on the client, even after removing the OBD_CONNECT_GRANT_PARAM conditional check, so it looks like some work is needed on the server as well.</p> <p>Johann Lombardi previously wrote, and I replied:</p> <blockquote> <p>Here is a short summary of the grant work status:</p> <p>1) the patch to estimate the overhead on the server-side (as discussed in Breckenridge) is landed. We have this in place for ldiskfs-osd.<br/> 2) While testing this patch, i have found that even for ldiskfs, the overhead estimate can sometimes be defeated, that's ORI-237. The problem is that we need a per-fragment overhead which only the client can add.</p></blockquote> <p>It makes sense to add some larger reservation even for ldiskfs, which can only be accessed by sync writes when the filesystem is nearly full. 1% is not unreasonable to reserve. Performance with a 99% full filesystem will already suck, so forcing the client to do sync RPCs is not any worse... It would be good if the client doing sync RPCs would still send a full-sized RPC if the write() syscall was large enough, rather than doing it 1 page at a time, but that is not related to grant, per se.</p> <blockquote> <p>3) While chatting with Alex &amp; Andreas, i realized that ZFS reports a blocksize of 1MB which is &gt; PAGE_SIZE. This causes a problem because the lustre client consumes PAGE_SIZE of grant space when dirtying one page, but this might end up consuming 1MB instead of 4KB on the backend filesystem (a factor of 256).</p></blockquote> <p>ext4 will soon also get 1MB+ block sizes, courtesy of Google. There is a feature called "bigalloc" which increases the "cluster" allocation unit to be a power-of-two multiple of the blocksize (still 4kB limited by PAGE_SIZE). This reduces overhead from bitmap searching/modification when doing large IOs, at the expense of wasting space within each cluster for smaller writes.</p> <p>The cluster size of the filesystem is fixed at format time and is constant for the whole filesystem, but that is fine for Lustre, since we already separate metadata from the data so we won't have problems with each directory consuming 1MB of space. The OST object directories will still consume space in 1MB chunks, but that is fine because we expect a million files to be created in each directory.</p> <blockquote> <p>4) The 1.8 &amp; 2.x client code has some basic support for blocksize &lt; PAGE_SIZE (it moves the over-consumed space to lost_grant). The problem is that this code goes mad if the server reports a blocksize &gt; PAGE_SIZE.</p> <p>To address problem #2, #3 &amp; #4, i am working on a patch set to implement the following:</p> <ul class="alternate" type="square"> <li>add a new connect flag OBD_CONNECT_LARGE_BSIZE. When this flag is set:</li> <li>the server reports the blocksize and the per-fragment overhead at connect time</li> </ul> </blockquote> <p>Is there an overhead beyond just rounding up the consumed grant to the blocksize? Do we want to take indirect, etc. blocks into account on the client, or is that entirely handled on the server in the grant overhead? The only other amount I can think of is the per-inode space consumption, which is 0 for ldiskfs (due to static allocation and write-in-place) but non-zero for COW filesystems like ZFS with CROW. If we are adding new connect fields, we may as well add both under a single OBD_CONNECT flag.</p> <p>Also, it seems that using 2 __u8 fields should be enough, since the blocksize and inode size is always going to be a power-of-two value, so sending the log2 value is enough, and that allows block/inode sizes up to 2^256 bytes in size. Since we are near the end of the fields in obd_connect_data that 1.8 can easily use, I'd prefer to use them sparingly for any feature that needs compatibility with 1.8 or 2.0. The &gt;2GB object size patch adds more space to obd_connect_data in a compatible way, but they will only be usable for patched clients, so I'd rather avoid that complexity if not needed.</p> <blockquote> <ul class="alternate" type="square"> <li>the client is able to consume grant more intelligently, taking into account the blocksize and the number of fragments (e.g. writting 256 contiguous pages consume 1MB + per-fragment overhead)</li> <li>when the client does not support this flag, there are 2 options (can be toogled through /proc):</li> <li>the default behavior is to emulate a 4KB blocksize (to address problem #4) and the server uses a grant inflation assuming that the client consumes 4KB of grant per OSD block (to address problem #3).<br/> OR</li> <li>the server refuses to grant space to those clients (disable writeback cache).</li> </ul> </blockquote> <p>I don't think Oleg assigned this to himself on purpose.</p> <p>Draft patch attached here: <a href="http://review.whamcloud.com/7793" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7793</a></p> <p>Here is the result for unpatched server(ldiskfs) and patched client, with sanity test_64b and sanityn test_15 enabled:<br/> I will change the test parameters and test other configurations and keep you updated.</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/ec4adf00-58eb-11e4-9a9c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/ec4adf00-58eb-11e4-9a9c-5254006e85c2</a></p> <p>result for unpatched server(zfs) and patched client:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/eba4dd82-59d4-11e4-8dbb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eba4dd82-59d4-11e4-8dbb-5254006e85c2</a></p> <p>Here is the for test only patch of patched server with unpatched client:<br/> <a href="http://review.whamcloud.com/#/c/12404/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12404/</a></p> <p>Johann Lombardi (johann.lombardi@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13531" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13531</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: delay grant releasing until commit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: edcd417f753864279a439ef8e80eff100fde3f72</p> <p>I will work on this.</p> <p>Jinshan, are you able to refresh this patch?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13531/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13531/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: delay grant releasing until commit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 31b7404f436241436fb0abdec2b6cd678c674d82</p> <p>Patch <a href="http://review.whamcloud.com/7793" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7793</a> needs to be refreshed and landed. </p> <p>The main goal of this patch is to reduce the grant over-provisioning for clients that do not understand large blocks on ZFS. It would be useful to run a manual test, or better to write a sanity subtest that compares the grant on the client with the grant on the server to ensure they roughly match rather than being inflated by a factor of (128/4).</p> <p>For ZFS OSTs the test should be skipped if this feature is not available on the OSC file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [ "$(facet_fstype ost1)" = "ZFS" ] &amp;&amp; $LCTL get_param osc.$FSNAME-OST0000*.import | grep -q "connect_flags:.*grant_param" || { skip "grant_param not available" &amp;&amp; return } </pre> </div></div> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/7793/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7793/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: add support for OBD_CONNECT_GRANT_PARAM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bd1e41672c974b97148b65115185a57ca4b7bbde</p> <p>I filed <a href="https://jira.whamcloud.com/browse/LU-7803" title="sanity test 78 failures in interop" class="issue-link" data-issue-key="LU-7803"><del>LU-7803</del></a> for a potential interop issue that I am experiencing now.</p> <p>It looks like everything has landed for this, can this bug be resolved?</p> <p>It doesn't appear that there was a test in the last patch to verify that the new grant code is working properly. I haven't looked in detail whether it is practical to make a test or not, but that should at least be given a few minutes attention before closing the bug. </p> <p>We started hitting the symptoms of <a href="https://jira.hpdd.intel.com/browse/LU-7510" class="external-link" target="_blank" rel="nofollow noopener">LU-7510</a> the last couple week or so, for which this patch is marked as a fix. We're running a 2.5-5 branch. Of our 80 OSTs, we had 32 that were ~90% full and the rest were closer to 65% full. Deactivating those fuller OSTs appears to have worked around the issue for now, though we think it's starting to happen on a sister file system.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21619" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21619</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> tests: FOR TEST ONLY GRANT_CHECK<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 64d3edb56be5b8af451a7b6947aad623fccf01ca</p> <p>After enabling grant checking:<br/> <a href="https://testing.hpdd.intel.com/test_sets/275100e8-5ff2-11e6-b5b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/275100e8-5ff2-11e6-b5b1-5254006e85c2</a></p> <p>All the tests that were checked failed. </p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25853" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25853</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: Fix grant interop with pre-GRANT_PARAM clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 15044478ce5f96a6bc80d8209e7fa9fed3f1a8a0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25853/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25853/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: Fix grant interop with pre-GRANT_PARAM clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 03f24e6f786459b3dd8a37ced7fb3842b864613d</p> <p>Landed for 2.10</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/21619/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21619/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> tests: Add GRANT_CHECK_LIST to sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 44c672e6aca39acbcca2aeb1b5b5b61a45265ce4</p> Blocker LU-3522 Duplicate LU-7970 LU-1507 Related LU-7510 LU-7803 LU-8007 Development Rank 1|hzv4lj: Rank (Obsolete) 4278