https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9413 Lustre <p>The issue has a detailed description in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=12739" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=12739</a></p> <p>In short, for an open file Lustre returns st_blksize=4 MiB and glibc allocates a buffer of this size. Short random reads cause 4 MiB BRWs which ruin performance as compared to other distributed fs.</p> <p>It is often not possible or not practical to patch the program itself.</p> <p>The original ticket contains the assertion that it's a bug in glibc and should be fixed in glibc. While it was fixed in glibc 2.25 (<a href="https://sourceware.org/bugzilla/show_bug.cgi?id=4099#c10" class="external-link" target="_blank" rel="nofollow noopener">https://sourceware.org/bugzilla/show_bug.cgi?id=4099#c10</a>), this glibc version won't be used in Linux distros with Lustre support any time soon.</p> <p>How about adding a temporary workaround for this issue similar to the one proposed by Aurélien Degrémont in bz #12739?</p> LU-9413

stat->st_blksize and glibc buffering

Bug Major Resolved Fixed WC Triage Andrew Perepechko patch Thu, 27 Apr 2017 16:00:27 +0000 Wed, 24 May 2017 12:23:00 +0000 Wed, 24 May 2017 12:23:00 +0000 Lustre 2.9.0 Lustre 2.10.0 0 5 <p>Andrew Perepechko (andrew.perepechko@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/26869" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26869</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9413" title="stat-&gt;st_blksize and glibc buffering" class="issue-link" data-issue-key="LU-9413"><del>LU-9413</del></a> llite: llite.stat_blocksize param for fixed st_blksize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 16964b5af530b59a3e7da73d62f3a8b1e0d8a4d0</p> <p>I'm glad you used a config param instead of a module parameter in the updated patch. However, as Aurelien said in Bugzilla, this parameter causes problems on the nodes that it is enabled on. Have you done any benchmarks with this enabled? I think iozone has a buffered IO mode with a range of blocksizes, with this feature enabled and disabled?</p> <p>As I also wrote in that patch:</p> <blockquote> <p>In Lustre 1.4 it was possible to change the stat() blocksize of a file by changing the file<br/> striping. The returned value was stripe_count * stripe_size, or 2MB, whichever was less.</p> <p>This was removed because NFS Connectathon failed an sanity test due to changing <tt>st_blksize</tt> if a file was <tt>mknod()</tt> (using an artifical <tt>st_blksize</tt> since the file has no layout) and then opened and it inherited a layout from the parent.</p></blockquote> <p>That said, I think the Connectathon issue could be fixed in several ways (e.g. set the <tt>st_blksize</tt> based on the filesystem default stripe size, rather than an arbitrary constant, or similar.</p> <p>Another option would be to use <tt>lfs ladvise</tt> to allow setting the blocksize on the file (in memory on the client inode, or possibly also on disk for ZFS <a href="https://jira.whamcloud.com/browse/LU-8951" title="lctl conf_param not retaining *_cache_enable settings" class="issue-link" data-issue-key="LU-8951"><del>LU-8951</del></a>). This would allow tuning the <tt>st_blksize</tt> value in a script before running the application, but it adds some complexity to run this on the file on every client for every time the application is running on a new file. Conceivably, <tt>st_blksize</tt> could be stored persistently in an xattr on the file (inherited from the parent?), but this is considerably more work.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26869/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26869/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9413" title="stat-&gt;st_blksize and glibc buffering" class="issue-link" data-issue-key="LU-9413"><del>LU-9413</del></a> llite: llite.stat_blocksize param for fixed st_blksize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f07576d3f82b50d74a858ccd60f7bdd0977a85a4</p> <p>Landed for 2.10</p> Development Epic/Theme Performance Rank 1|hzzbd3: Rank (Obsolete) 9223372036854775807 Severity