https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11264 Lustre <p>I'm going to bug Cray about this but Andreas said  "we can't fix problems that aren't in Jira" so here I am.</p> <p>In the old days one would get striping information via an ioctl:</p> <p>   </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct lov_user_md *lum;     size_t lumsize = sizeof(struct lov_user_md) +LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data);     lum-&gt;lmm_magic = LOV_USER_MAGIC;     ret = ioctl( fd, LL_IOC_LOV_GETSTRIPE, (void *)lum ); </pre> </div></div> <p> </p> <p> </p> <p>Then the llapi_ routines showed up:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">    ret = llapi_file_get_stripe(opt_file, lum);</pre> </div></div> <p>and even more sophisticated llapi_layout_ routines:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct llapi_layout *layout; layout = llapi_layout_get_by_fd(fd, 0); llapi_layout_stripe_count_get(layout, &amp;(lum-&gt;lmm_stripe_count)); llapi_layout_stripe_size_get(layout, &amp;(lum-&gt;lmm_stripe_size)); llapi_layout_ost_index_get(layout, 0, &amp;(lum-&gt;lmm_stripe_offset));   </pre> </div></div> <p> </p> <p>I haven't done a full scaling run, but small scale tests don't look great for the newer interfaces.   Average time to get the stripe size for a 2 node 128 process MPI job:</p> <p>ioctl:   17.2 ms</p> <p>llapi_file_get_stripe: 142.0 ms</p> <p>llapi_layout:  2220 ms</p> <p> </p> <p>Wow!  I want to use the new interfaces but these overheads are bonkers.</p> <p> </p> Argonne&#39;s Theta machine is a Cray running lustre: 2.7.3.74 <br/> kernel: patchless_client <br/> build: 2.7.3.74 LU-11264

llapi_* routines demonstrate poor performance

Improvement Major Open Unresolved WC Triage Robert Latham Fri, 17 Aug 2018 15:23:43 +0000 Thu, 14 Sep 2023 04:50:19 +0000 0 6 <p>I attached a graph of a scaling experiment on NERSC's Cray Lustre deployment on Cori from I think two years ago.  The problem is not a recent regression.</p> <p>Robert,</p> <p>I'm a Cray person, but I'll discuss this here so as not to leave the Whamcloud guys in the cold.</p> <p>What exactly are you measuring?  Is this execution time of the function directly, within a binary, or is it execution time of an entire binary containing just this function call?</p> <p>Can you share whatever your test code is here?  Just to make our lives easier.</p> <p>A few more thoughts/questions.</p> <ol> <li>The llapi_file_get_stripe function you selected is <b>not</b> a direct equivalent of that ioctl.  It also has to open() the file.  So that may be a source of some of the overhead.</li> <li>Which of the llapi_layout functions is taking that much time?  You don't specify.</li> </ol> <p>Those last three functions (stripe_count_get, stripe_size_get, and ost_index_get) literally just access pointers in the memory given, so if they're taking milliseconds by themselves...</p> <p> </p> <p>Also, can you compare execution time from one thread?  If it's really as bad as your measurements indicate, it should be straightforward to do so.</p> <p>I attached the test program I used.  it's testing a bunch of different things but to get the routines in this test case:</p> <ul> <li>ioctl to get information: llapi-perf -i LUSTRE_FILE</li> <li>llapi_file_get_stripe : llapi-perf -a LUSTRE_FILE</li> <li>llapi_layout_get_by_fd: llapi_perf -A LUSTRE_FILE</li> </ul> <p>Absolutely agreed that llapi_file_get_stripe should be expensive since it is operating on a filename. Was astonished to see llapi_layout aproach took so long considering it operates on the fd directly.</p> <p>I assumed the initial llapi_layout_get_by_fd call took all the time, but I was measuring the time to call the four routines together &#8211; after all, there's no reason for an application to just call get_by_fd. <br/> I'll add some finer grained timing information and re-run the experiment<br/>  </p> <p>Thanks, Rob - I actually just grabbed your test and did a bunch of testing myself.</p> <p>I see a really wildly large gap even with 1 process - interesting.</p> <p>This is with debug turned on, on a small VM setup:</p> <p><span class="error">&#91;root@cent7c01 cent7s01&#93;</span># rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -i testfile; lctl dk &gt; /tmp/out3<br/> opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 0, opt_ioctl: 1, opt_fpp: 0, opt_llapi_fd: 0, nprocs: 1, time: 2.622000 ms</p> <p><span class="error">&#91;root@cent7c01 cent7s01&#93;</span># rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -a testfile; lctl dk &gt; /tmp/out<br/> opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 1, opt_ioctl: 0, opt_fpp: 0, opt_llapi_fd: 0, nprocs: 1, time: 3.715000 ms<br/> <span class="error">&#91;root@cent7c01 cent7s01&#93;</span># rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -A testfile; lctl dk &gt; /tmp/out2<br/> opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 0, opt_ioctl: 0, opt_fpp: 0, opt_llapi_fd: 1, nprocs: 1, time: 5.026000 ms</p> <p>So that's "ioctl" at 2.6 ms</p> <p>Get_stripe at 3.7 ms<br/> And<br/> get_by_fd at 5.0 ms</p> <p>That's... wild.  I'll try to dig in later.</p> <p>EDIT:<br/> My previous comment about no additional server communications was wrong; had an error in my data collection.</p> <p>The simple create+ioctl does 2, one of which is close().</p> <p>The llapi_get_stripe function does 3 (create, some sort of get, then close).</p> <p>The layout get function does 4<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/warning.png" height="16" width="16" align="absmiddle" alt="" border="0"/>.</p> <p>I've got some ideas why - I'll take a further look next week.</p> <p>oh yeah, lots of variance in these tests. It's not statistically rigorous but above numbers came from me running each test 5 times and reporting an average.</p> <p>Robert,</p> <p>Here's the problem in a nutshell.</p> <p>lfs getstripe asks the client for the stripe count information that it has locally, and gets it locally if it's present.  This is fast because that information is generally already cached locally when the file is open.</p> <p>The other two functions do not ask the client for the layout, as such - Instead, the ioctl they use actually directly reads the layout extended attribute from the MDS.  This read of an extended attribute cannot be satisfied by the layout information held in memory on the client (which it usually has if the file is open), instead, it goes directly to the MDS.</p> <p>The solution would be to rewrite the implementation of those new ioctls to get the layout that the client has in memory and request it if it's not present.</p> <p>By the way, thanks for your quick reply last week.</p> <p>I just happened to be looking at this code on the <tt>setstripe</tt> side and see a number of places where it could be optimized. Running a simple test comparing <tt>lfs setstripe</tt> with <tt>touch</tt> and <tt>createmany</tt> shows a big difference in performance (files removed between each test):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ time lfs setstripe -c 1 /myth/tmp/tmp/f{1..1000} : real: 0m11.089s, user: 0m0.115s, sys: 0m2.132s $ time touch /myth/tmp/tmp/f{1..1000} : real: 0m6.400s, user: 0m0.018s, sys: 0m0.863s $ time createmany -o /myth/tmp/tmp/f 1000 : real: 0m2.329s, user: 0m0.008s, sys: 0m0.185s </pre> </div></div> <p>It is pretty clear from the elapsed time and the system time that <tt>lfs setstripe</tt> is doing a lot more stuff for each file create compared to the others (<tt>touch</tt> is additionally setting the timestamps). Looking at the <tt>strace</tt> of <tt>lfs setstripe -c1</tt> on a client shows what is going on:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>gettid() = 3237 open("/dev/urandom", O_RDONLY|O_NOFOLLOW) = 3 read(3, "Fg#\306", 4) = 4 close(3) = 0 lstat("/myth", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0 lstat("/myth/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 lstat("/myth/tmp/tmp", {st_mode=S_IFDIR|0775, st_size=53248, ...}) = 0 lstat("/myth/tmp/tmp/foo", 0x7ffe9dc99410) = -1 ENOENT (No such file or directory) lstat("/myth", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0 lstat("/myth/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 lstat("/myth/tmp/tmp", {st_mode=S_IFDIR|0775, st_size=53248, ...}) = 0 lstat("/myth/tmp/tmp/foo", 0x7ffe9dc99410) = -1 ENOENT (No such file or directory) lstat("/myth", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0 lstat("/myth/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 lstat("/myth/tmp/tmp", {st_mode=S_IFDIR|0775, st_size=53248, ...}) = 0 open("/proc/mounts", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 read(3, "rootfs / rootfs rw 0 0\nproc /pro"..., 1024) = 1024 read(3, "tmp ext4 ro,relatime,barrier=1,d"..., 1024) = 88 read(3, "", 1024) = 0 close(3) = 0 open("/myth/tmp/tmp/foo", O_WRONLY|O_CREAT|O_LOV_DELAY_CREATE, 0644) = 3 ioctl(3, LL_IOC_LOV_SETSTRIPE, 0x942060) = 0 </pre> </div></div> <p>Firstly, it appears that <tt>liblustreapi_init()</tt> function is always doing some work to initialize the PRNG, but this PRNG is only used in a small number of cases. This initialization can be skipped unless the PRNG is actually needed in particular functions, even though it is only a one-time thing it is still needless overhead.</p> <p>More importantly, it is doing multiple traversals of the full pathname (bad) to do <tt>realpath()</tt> on the filename and processing <tt>/proc/mounts</tt> for each file to determine if the file is on a Lustre filesystem or not. IMHO, it should be enough to return an error from <tt>ioctl(LL_IOC_LOV_SETSTRIPE)</tt> to determine that the file is not on a Lustre filesystem, or if that is really deemed necessary we can do <tt>statfs()</tt> on the directory and check the magic without doing dozens of syscalls.</p> <p>There is also extra overhead if a pool name is specified to verify that it is valid that could be a one-time check for repeated use of the same pool(s), and to verify the <tt>stripe_size &gt;= PAGE_SIZE</tt>, where again the <tt>PAGE_SIZE</tt> does not change from one file to the next.</p> <p>Finally, I observe in the debug logs that between <tt>open(O_LOV_DELAY_CREATE)</tt> and <tt>ioctl(LL_IOC_LOV_SETSTRIPE)</tt> there is a DLM lock callback from the MDS to the client for the layout  lock (I think). It doesn't make sense to grant the <tt>MDS_INODELOCK_LAYOUT</tt> lock to the client opening with <tt>O_LOV_DELAY_CREATE</tt>, when we know the client will immediately be changing it.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35028" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35028</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11264" title="llapi_* routines demonstrate poor performance" class="issue-link" data-issue-key="LU-11264">LU-11264</a> utils: avoid pool overhead in lfs setstripe<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e1246538418d2b58f5457d91956c7c75a9fb1153</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35091" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35091</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11264" title="llapi_* routines demonstrate poor performance" class="issue-link" data-issue-key="LU-11264">LU-11264</a> llapi: reduce llapi_stripe_limit_check() overhead<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dfea5bd239531a91d19f603945601e39f9254f84</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35091/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35091/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11264" title="llapi_* routines demonstrate poor performance" class="issue-link" data-issue-key="LU-11264">LU-11264</a> llapi: reduce llapi_stripe_limit_check() overhead<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1c8ce141c6b6b621cf0fc89d32e1e7a2db43729d</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35092/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35092/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11264" title="llapi_* routines demonstrate poor performance" class="issue-link" data-issue-key="LU-11264">LU-11264</a> llapi: clean up llapi_search_tgt() code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 096db80e0810b4abb8e268a58623c23bf0b57d2c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35093/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35093/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11264" title="llapi_* routines demonstrate poor performance" class="issue-link" data-issue-key="LU-11264">LU-11264</a> llapi: clean up llapi_file_create_foreign()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: efe6615d7558ee6bda106546c55d92758bec528d</p> Duplicate Related LU-14645 LU-13693 LU-17025 LU-14316 LU-14897 LU-10500 LU-12327 LU-12369 LU-14396 Development Epic/Theme Performance Rank 1|i000y7: Rank (Obsolete) 9223372036854775807