https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14055 Lustre <p><a href="https://review.whamcloud.com/37834" class="external-link" target="_blank" rel="nofollow noopener">commit 76626d6c52</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> all: Separate debugfs and procfs handling</tt>" caused write performance regression. Here is a reproducer and tested workload.</p> <p>Single Client(Ubuntu 18.04, 5.4.0-47-generic), 16MB O_DIRECT, FPP (128 processes)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun --allow-run-as-root -np 128 --oversubscribe --mca btl_openib_warn_default_gid_prefix 0 --bind-to none ior -u -w -r -k -e -F -t 16384k -b 16384k -s 1000 -u -o /mnt/ai400x/ior.out/file --posix.odirect </pre> </div></div> <p>"git bisect" indentified an commit where regression started.</p> <p>Here is test results.<br/> 76626d6c52 <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> all: Separate debugfs and procfs handling</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 21861 1366.33 60.78 16384 16384 0.091573 93.68 40.38 93.68 0 read 38547 2409.18 46.14 16384 16384 0.005706 53.13 8.26 53.13 0 </pre> </div></div> <p>5bc1fe092c <a href="https://jira.whamcloud.com/browse/LU-13196" title="Remove inode mutex for DIO reads" class="issue-link" data-issue-key="LU-13196"><del>LU-13196</del></a> llite: Remove mutex on dio read</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 32678 2042.40 58.96 16384 16384 0.105843 62.67 4.98 62.67 0 read 38588 2411.78 45.89 16384 16384 0.004074 53.07 8.11 53.07 0 </pre> </div></div> <p>master (commit 56526a90ae)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 17046 1065.37 119.02 16384 16384 0.084449 120.15 67.76 120.15 0 read 38512 2407.00 45.04 16384 16384 0.006462 53.18 9.07 53.18 0 </pre> </div></div> <p>master still has this regression and when commit 76626d6c52 reverts from master, the performrance is back.</p> <p>master (commit 56526a90ae)+ revert commit 76626d6c52</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 32425 2026.59 59.88 16384 16384 0.095842 63.16 4.79 63.16 0 read 39601 2475.09 47.22 16384 16384 0.003637 51.72 5.73 51.72 0 </pre> </div></div> master (commit: 56526a90ae) LU-14055

Write performance regression caused by an commit from LU-13344

Bug Major Resolved Fixed Patrick Farrell Shuichi Ihara Wed, 21 Oct 2020 04:12:56 +0000 Wed, 5 Jan 2022 19:51:08 +0000 Wed, 5 Jan 2022 19:51:08 +0000 Lustre 2.14.0 Lustre 2.15.0 0 20 <p>I looked through that patch and didn't see anything obvious that would introduce a performance regression, since none of those proc/seq files should be anywhere close to the IO path. I think the likely source of the problem is that the change from <tt>/proc</tt> to <tt>/sys/kernel/debug</tt> has resulted in some tunable parameter to be lost completely (random example <tt>readcache_max_io_mb</tt>), or is somehow "disconnected" from the actual parameter (e.g. the parameter can be set, but it doesn't actually change the behavior of the kernel).</p> <p>There weren't any parameters that I could easily see, beyond the <tt>readcache_max_io_mb</tt> and related <tt>osd-ldiskfs</tt> parameters that could be the culprit.</p> <p>it might be related to what cpu is quite busy rather than IO? A test case was 128 IOR process against 128 CPU cores and client's cpu was quite busy and I saw some ior threads were much slower than others in bad case. Let me dig in this detail.</p> <p>I don't think <em>any</em> of the code changed by the patch would actually be used during operation. Are there any of tunable parameters set on the client or server that were modified by this patch? I think the most likely cause is that one of the modified parameter settings is not working properly. It might be possible to revert part (ideally half) of the parameter changes to see if that avoids or reintroduces the problem, to isolate it down to a specific part of the patch?</p> <p>I only reverted patch on client and performance was back.<br/> Also, what parameters applied was below. No other parameters applied on both server and client.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>obdfilter.*.brw_size: 16 osd-ldiskfs.*.writethrough_cache_enable: 0 osd-ldiskfs.*.read_cache_enable: 0 </pre> </div></div> <p>Any thoughts on this one? Should we just revert this patch while we are working out the details?</p> <p>I looked through the patch again and I do not see anything obvious.<br/> It would be fine to revert this for now.</p> <p>I can't reproduce this problem. Is it only a Ubunut issue? Can you reproduce it Shaun? Does it break mpirun or mdtest / IOR. Lustre is only used by mdtest to call llapi_dir_set_default_lmv_stripe() for the -g option. What setup do you have ? Everything on one node or all VMs?</p> <p>No I don't have a reproducer here. My ubuntu testing is just a couple of VMs.<br/> If it is 5.3/4 kernel related and not specific to Ubuntu then I expect I will have something available soonish but not immediately.</p> <p>Shaun, would you be able to post a handful of different patches that revert different parts of patch <a href="https://review.whamcloud.com/37834" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37834</a> to see if Ihara can isolate which part is slowing things down?</p> <p>Sounds good. I will post a series of reverts.</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40833</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Enable debugfs macros fallback to PDE_DATA<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6ef6b0b9b9ee670930a7a1863f4903f783f58055</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40835" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40835</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 1 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3a42232a0a536758121cd1ebcb4c73334d1d898f</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40836" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40836</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 2 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e2c1be578c090153aaaa752ac8bad2f6668923d2</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40837" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40837</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 3 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a650c8a62227589adc940b53a0432d28a4276274</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40838" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40838</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 4 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6b994fbaf5e0b7a6eef7266c6f9b0836115cbad</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40839" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40839</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 5 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a8b7b7877beea237164312f5581fe21919df87b9</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40840" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40840</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 6 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 43fdb7aa30367545452d242e41b2b95fbd6832f4</p> <p>Shaun Tancheff (shaun.tancheff@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40841" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40841</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> all: Revert <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> 7 of 7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ca7e834e5472c520b8a4443fd13d44fdacd230a6</p> <p>Shuichi, could you please give these patches a try to help isolate the source of the regression. </p> <p>Shuichi I pushed a fixed patch set that Shaun started. Please give it a try. Some of the RHEL8.2 builds failed due to an unrelated bug. Those patches do build as you can see in later patches building properly.</p> <p>Thank you James! I will test new patches on my test node and let you know.</p> <p>Here is test results after reverted patch. ("Write" performance only measured) So, it still needed to revert all patches to get the performance back.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>7 of 7 patch 29,406 MiB/s 6 of 7 patch 18,791 MiB/s 5 of 7 patch 18,924 MiB/s 4 of 7 patch 18,715 MiB/s </pre> </div></div> <p>I didn't test 3,2,1 of 7 patch since regression already started when 6 of 7 patch reverted. should we split a bit 7 of 7 patch to narrow down?</p> <p>Shuichi, just to confirm, the "good" number beside "7 of 7" is with <b>all 7</b> of the patches applied (ie. the entire original patch reverted), and if the "7 of 7" patch is removed then the performance is bad?</p> <p>What is strange is that the "7 of 7" patch is not really doing much except renaming a few structs and adding <b>one</b> variable near the end of <tt>struct obd_device</tt>. I had previously wondered if the increase in <tt>struct obd_device</tt> size was causing something bad with cache lines on SMP systems, or the struct became too large for some reason (it is just over 4KB now)?</p> <p>You could test this by reverting all of the patches, the only adding a "<tt>void *dummy;</tt>" pointer where "<tt>struct ldebugfs_vars *obd_debugfs_vars</tt>" is located in the "7 of 7" patch in <a href="https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h</a></p> <p>Other than that, I can't see anything in the "7 of 7" patch that would cause any problems. The /proc files changed are for "fid" and "fld" services, which I don't think are being tuned at all, so it shouldn't matter even if the tunables were broken. </p> <blockquote> <p>Shuichi, just to confirm, the "good" number beside "7 of 7" is with all 7 of the patches applied (ie. the entire original patch reverted), and if the "7 of 7" patch is removed then the performance is bad?</p></blockquote> <p>That's correct. Once patch "7 of 7" applied (reverted entire <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> patch), the performance was good.</p> <blockquote> <p>You could test this by reverting all of the patches, the only adding a "void *dummy;" pointer where "struct ldebugfs_vars *obd_debugfs_vars" is located in the "7 of 7" patch in <a href="https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h</a></p></blockquote> <p>okey, I will test this idea.</p> <p>I noticed that even before the problematic patch is applied, there are already some places in the code using "<tt>ldebugfs</tt>" that are affected by the changes in the patch. Most places are <tt>fid</tt>, <tt>fld</tt>, <tt>seq</tt>, which I don't think can be harmful. There is a bit of usage in <tt>ldlm</tt>, but it doesn't look harmful either. </p> <p>One important parameter that may be affected is <tt>llite.&#42;.max_cached_mb</tt>". </p> <p>Shuichi, the other thing you could try with and without the patch applied is "<tt>lctl get_param -R '*'</tt>" to see if the parameters available changes between the two versions?</p> <p>For the test runs, was there a default striping set on the filesystem?</p> <blockquote> <p>For the test runs, was there a default striping set on the filesystem?</p></blockquote> <p>yes, stripe count was 1 for all tests.</p> <blockquote><p>You could test this by reverting all of the patches, the only adding a "void *dummy;" pointer where "struct ldebugfs_vars *obd_debugfs_vars" is located in the "7 of 7" patch in <a href="https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/40841/2/lustre/include/obd.h</a></p></blockquote> <p>Andreas, here is test resutls of your suggestion. And it seems that results tells us a root cause that you exaclty pointed out. it was tested 3 times to verify.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Write(MB/s)</th> <th class='confluenceTh'>"7 of 7" revert patch</th> <th class='confluenceTh'>"7 of 7" revert patch + adding "struct ldebugfs_vars *obd_debugfs_vars"</th> </tr> <tr> <td class='confluenceTd'>iteration1</td> <td class='confluenceTd'>36.225</td> <td class='confluenceTd'>19.280</td> </tr> <tr> <td class='confluenceTd'>iteration2</td> <td class='confluenceTd'>35.410</td> <td class='confluenceTd'>18.781</td> </tr> <tr> <td class='confluenceTd'>iteration3</td> <td class='confluenceTd'>33.871</td> <td class='confluenceTd'>19.033</td> </tr> </tbody></table> </div> <p>If it isn't too much trouble (and if you didn't already) can you try adding that field at the <b>end</b> of the structure, rather than where it currently is?</p> <p>If that makes a difference, then my guess is that there is a hot cache-line in that struct which is being split.</p> <p>If it doesn't make a difference, then my guess is that the struct size crosses a boundary  resulting in obd_device_cache being shared with a different other cache, which maybe doesn't something strange.  I doubt that though.</p> <p>I should have to inform that adding a "void *dummy;" top of "7 of 7" revert patch also caused same performance problem, then tested adding "struct ldebugfs_vars *obd_debugfs_vars" after of that. </p> <p>Very strange that adding an 8-byte pointer to the already-large <tt>struct obd_device</tt> causes such a huge performance problem. According to <tt>pahole</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* size: 6808, cachelines: 107, members: 98 */</span> <span class="code-comment">/* sum members: 6792, holes: 2, sum holes: 8 */</span> <span class="code-comment">/* sum bitfield members: 20 bits, bit holes: 1, sum bit holes: 44 bits */</span> <span class="code-comment">/* paddings: 2, sum paddings: 8 */</span> <span class="code-comment">/* forced alignments: 1 */</span> <span class="code-comment">/* last cacheline: 24 bytes */</span> } __attribute__((__aligned__(8))); </pre> </div></div> <p>Maybe it pushes the struct from 106 cachelines to 107, but I wouldn't think that would make a <em>huge</em> difference in performance. We <em>could</em> pack the two 4-byte holes in the struct to get back 8 bytes to solve this problem quickly before 2.14. That means the <em>next</em> patch that touches this struct is going to cause problems again, but would also give us some breathing room to resolve this issue more completely in 2.15. I will make a patch for this.</p> <p>pahole shows a number of other major offenders in the struct that could be cleaned up:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct rhashtable obd_uuid_hash; <span class="code-comment">/* 200 176 */</span> struct rhltable obd_nid_hash; <span class="code-comment">/* 376 176 */</span> struct obd_llog_group obd_olg; <span class="code-comment">/* 936 176 */</span> struct hrtimer obd_recovery_timer __attribute__((__aligned__ (8))); <span class="code-comment">/* 1272 80 */</span> union { struct obd_device_target obt; <span class="code-comment">/* 1520 96 */</span> struct filter_obd filter; <span class="code-comment">/* 1520 96 */</span> struct ost_obd ost; <span class="code-comment">/* 1520 72 */</span> struct echo_obd echo; <span class="code-comment">/* 1520 336 */</span> struct client_obd cli; <span class="code-comment">/* 1520 2664 */</span> struct echo_client_obd echo_client; <span class="code-comment">/* 1520 56 */</span> struct lov_obd lov; <span class="code-comment">/* 1520 680 */</span> struct lmv_obd lmv; <span class="code-comment">/* 1520 4872 */</span> } u; <span class="code-comment">/* 1520 4872 */</span> </pre> </div></div> <p>There seems to have been a <b>lot</b> of cruft added to <tt>obd_device</tt> over the years that is server-specific and could be moved into <tt>struct obd_device_target</tt> instead of being kept in the common struct:</p> <ul class="alternate" type="square"> <li><tt>obd_nid_hash</tt> <b>usage</b> is already under <tt>HAVE_SERVER_SUPPORT</tt> and could be declared that way also.  I have a patch for this.  Moving it into <tt>obd_device_target</tt> would be better since it would also reduce memory usage on clients that are built with server support.</li> <li><tt>obd_uuid_hash</tt> entry is also only needed on the server. While it is referenced by <tt>__class_new_export()</tt> (also used on the client for "self" exports), it should only really be needed for targets that add remote connections. The whole block that checks <tt>obd_uuid_equals(cluuid, obd_uuid)</tt> could be under <tt>HAVE_SERVER_SUPPORT</tt> to allow <tt>obd_uuid_hash</tt> to also be moved into <tt>obd_device_target</tt>, along with <tt>obd_uuid_add()</tt> and <tt>obd_uuid_del()</tt>. I have a patch for this.</li> <li><tt>obd_olg</tt> is larger than it needs to be, since many of the <tt>LLOG&#95;&#42;&#95;CTXT</tt> contexts are no longer used (at least <tt>LLOG_MDS_OST_REPL_CTXT</tt>, <tt>LLOG_SIZE_ORIG,REPL_CTXT</tt>, and <tt>LLOG_TEST_REPL_CTXT</tt>. <a href="https://jira.whamcloud.com/browse/LU-5218" title="Interop 2.5.1&lt;-&gt;2.6 failure on test suite lustre-rsync-test test_1: ASSERTION( index &gt;= 0 &amp;&amp; index &lt; LLOG_MAX_CTXTS ) failed" class="issue-link" data-issue-key="LU-5218"><del>LU-5218</del></a> explains some of the details. <tt>OST_SZ_REC</tt> which <em>seems</em> to imply that <tt>LLOG_SIZE_ORIG_CTXT</tt> is used is itself not actually <b>set</b> anywhere but test code, and was never used in production (it was for an old Size-on-MDS implementation that was never finished). Removing the unused <tt>LLOG_&#42;_CTXT</tt> constants will not directly shrink <tt>LLOG_MAX_CTXTS</tt>, but may allow a more efficient mapping to be used (e.g. a simple mapping to an in-memory dense enum).</li> <li><tt>obd_recovery_timer</tt> is only needed on the server and can move into <tt>obd_device_target</tt>, along with all of the other <tt>obd&#95;recovery&#95;&#42;</tt>, <tt>obd&#95;replay&#95;&#42;</tt>, <tt>obd_lwp_export</tt>, <tt>obd_exports_timed</tt>, <tt>obd_eviction_timer</tt>, <tt>obd&#95;&#42;&#95;clients</tt>, and at least some of the <tt>obd&#95;&#42;transno&#42;</tt> fields (though clients may use some of them). The <tt>target&#95;&#42;()</tt> functions under <tt>HAVE_SERVER_SUPPORT</tt> in <tt>ldlm_lib.c</tt> should all be moved into <tt>lustre/target/tgt_recovery.c</tt> (or similar) since they don't really have anything to do with LDLM.</li> </ul> <p>The worst offender is the device-specific union <tt>u</tt>, with <tt>client_obd</tt> and <tt>lmv_obd</tt> being the largest members (though <tt>obd_device_target</tt> may increase in size in the future). One option is to dynamically allocate this member depending on the type used, since there is typically only one <tt>lmv_obd</tt> on a client, though <tt>client_obd</tt> is used for most of the devices on the client so will not help much.</p> <p>The (almost only) offender in <tt>lmv_obd</tt> is <tt>struct lu_tgt_descs lmv_mdt_descs</tt>, which is a static array for all of the potential MDT devices the LMV may have (<tt>struct lu_tgt_desc_idx *ltd_tgt_idx<span class="error">&#91;TGT_PTRS&#93;</span></tt> being the major contributor). This is really unnecessary, and could be dynamically allocated for the maximum current MDT count. I have included that into patch <a href="https://review.whamcloud.com/40901" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40901</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13601" title="page allocation failure during mount " class="issue-link" data-issue-key="LU-13601"><del>LU-13601</del></a> llite: avoid needless large allocations</tt>" but it deserves to be split into its own patch, since making <tt>ltd_tgt_index</tt> dynamically sized is relatively easy compared to the llite changes also in that patch. I have a patch for this already.</p> <p>The worst offender in <tt>client_obd</tt> are the seven <tt>obd_histogram</tt> fields, consuming 1848 of 2664 bytes, with 34 bytes of holes. I had a patch to dynamically allocate these structures on an as-needed basis, but it was complex and never landed. Maybe I need to revive that patch again.</p> <p>As for fields that might be affected by the addition of <tt>obd_debugfs_vars</tt> there are only a few that are after this new field, and of those only a subset are used on the client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* --- cacheline 101 boundary (6464 bytes) --- */</span> struct ldebugfs_vars * obd_debugfs_vars; <span class="code-comment">/* 6464 8 */</span> atomic_t obd_evict_inprogress; <span class="code-comment">/* 6472 4 */</span> <span class="code-comment">/* XXX 4 bytes hole, <span class="code-keyword">try</span> to pack */</span> wait_queue_head_t obd_evict_inprogress_waitq; <span class="code-comment">/* 6480 24 */</span> struct list_head obd_evict_list; <span class="code-comment">/* 6504 16 */</span> rwlock_t obd_pool_lock; <span class="code-comment">/* 6520 8 */</span> <span class="code-comment">/* --- cacheline 102 boundary (6528 bytes) --- */</span> __u64 obd_pool_slv; <span class="code-comment">/* 6528 8 */</span> <span class="code-object">int</span> obd_pool_limit; <span class="code-comment">/* 6536 4 */</span> <span class="code-object">int</span> obd_conn_inprogress; <span class="code-comment">/* 6540 4 */</span> struct lu_ref obd_reference; <span class="code-comment">/* 6544 0 */</span> struct kset obd_kset; <span class="code-comment">/* 6544 160 */</span> <span class="code-comment">/* --- cacheline 104 boundary (6656 bytes) was 48 bytes ago --- */</span> struct kobj_type obd_ktype; <span class="code-comment">/* 6704 72 */</span> <span class="code-comment">/* --- cacheline 105 boundary (6720 bytes) was 56 bytes ago --- */</span> struct completion obd_kobj_unregister; <span class="code-comment">/* 6776 32 */</span> </pre> </div></div> <p>Likely candidates might be <tt>obd_pool_slv</tt> being split from <tt>obd_pool_lock</tt>, or something in the recently-added <tt>obd_kset</tt> or <tt>obd_ktype</tt> fields. I'm hoping that shrinking <tt>obd_device</tt> will resolve the problem, since random field alignment issues causing such huge performance swings is a nightmare.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41161" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41161</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> obdclass: fill hole in struct obd_device<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 29898c50da924ad146b71046ee066371058cbb6a</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41162" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41162</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> lmv: reduce struct lmv_obd size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4f6159a3f2175cdcb8eee4017ba8e0a3d70268f2</p> <p>Shuichi, could you please try with these two patches. The first one just decreases the struct size by 4 bytes, but I'm not sure if that will be enough. The second reduces it by over 2KB, which hopefully is enough. Otherwise, it may be that there is an alignment issue with some struct (likely the new "<tt>obd_k&#42;</tt>" ones at the end) that will need some specific alignment requests. </p> <p>I cannot tell if this is already understood but it would be useful to know if the change in performance is due to the debugfs changes being applied to the client, to the server, or both.</p> <p>John, it is on the client only, AFAIK. </p> <p>Shuichi, could you please run "<tt>pahole</tt>" on the obdclass.ko module with and without the problem and attach it here. That program is part of the "<tt>dwarves</tt> RPM. </p> <p>Andreas, <br/> I just tested two patches, but the performrance was even worse below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 14218 888.67 161.01 16384 16384 0.159725 172.84 27.90 172.85 0 write 14168 885.51 159.56 16384 16384 0.218456 173.46 23.85 173.46 0 write 14093 880.82 161.54 16384 16384 0.191401 174.38 25.50 174.38 0  </pre> </div></div> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41178" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41178</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> obdclass: move obd_debugfs_vars to end obd_device<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dd0a0df383387e6455bbad565503883433516454</p> <p>Shuichi, if shrinking <tt>struct obd_device</tt> does not solve the problem, then it seems the problem is caused by a misalignment of some data structure that follows the added <tt>obd_debugfs_vars</tt> field.</p> <p>Can you please try another set of tests that move the "<tt>obd_debugfs_vars</tt>" line until we isolate the problematic field. The first test would be to move <tt>obd_debugfs_vars</tt> to the end of the struct:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* --- cacheline 101 boundary (6464 bytes) --- */</span> - struct ldebugfs_vars * obd_debugfs_vars; <span class="code-comment">/* 6464 8 */</span> atomic_t obd_evict_inprogress; <span class="code-comment">/* 6472 4 */</span> wait_queue_head_t obd_evict_inprogress_waitq; <span class="code-comment">/* 6480 24 */</span> struct list_head obd_evict_list; <span class="code-comment">/* 6504 16 */</span> rwlock_t obd_pool_lock; <span class="code-comment">/* 6520 8 */</span> <span class="code-comment">/* --- cacheline 102 boundary (6528 bytes) --- */</span> __u64 obd_pool_slv; <span class="code-comment">/* 6528 8 */</span> <span class="code-object">int</span> obd_pool_limit; <span class="code-comment">/* 6536 4 */</span> <span class="code-object">int</span> obd_conn_inprogress; <span class="code-comment">/* 6540 4 */</span> struct lu_ref obd_reference; <span class="code-comment">/* 6544 0 */</span> struct kset obd_kset; <span class="code-comment">/* 6544 160 */</span> <span class="code-comment">/* --- cacheline 104 boundary (6656 bytes) was 48 bytes ago --- */</span> struct kobj_type obd_ktype; <span class="code-comment">/* 6704 72 */</span> <span class="code-comment">/* --- cacheline 105 boundary (6720 bytes) was 56 bytes ago --- */</span> struct completion obd_kobj_unregister; <span class="code-comment">/* 6776 32 */</span> + struct ldebugfs_vars * obd_debugfs_vars; <span class="code-comment">/* 6464 8 */</span> </pre> </div></div> <p>to see if this solves the problem (without my other patches). I've pushed a patch to do this. If it fixes the problem, then this confirms that the problem is caused by the alignment or cacheline contention on of one of the fields between <tt>obd_evict_inprogress</tt> and <tt>obd_kobj_unregister</tt>. This would be enough to land for 2.14.0 to solve the problem, but I don't want to leave the reason for the problem unsolved, since it is likely to be accidentally returned again in the future (e.g. by landing my patches to shrink <tt>lu_tgt_desc</tt> or anything else).</p> <p>To isolate the reason for the problem you would need to "bisect" the 11 fields/366 bytes to see which one is causing the slowdown.</p> <p>First try moving <tt>obd_debugfs_vars</tt> after <tt>obd_kset</tt> to see if this causes the slowdown again. If not, then the problem is <tt>obd_kset</tt> or earlier, so try moving it immediately before <tt>obd_kset</tt> (this is the largest field so makes it difficult to "bisect" exactly). If the problem is still not seen, move it after <tt>obd_evict_list</tt>, etc. Essentially, when <tt>obd_debugfs_vars</tt> is immediately <em>before</em> the offending struct the performance will be bad, and when it is immediately <em>after</em> the struct then the performance problem should go away. Once you find out what the structure is, try moving that field to be at the start of <tt>struct obd_device</tt> so that there is no chance of it being misaligned, after <tt>obd_lu_dev</tt> and after <tt>obd_recovery_expired</tt>. If these also show good performance, then this can be a permanent solution (I would prefer after <tt>obd_recovery_expired</tt> since these bitfields are very commonly used).</p> <p>Please run the "<tt>pahole</tt>" command on the <tt>obdclass.ko</tt> module to show the "good" and "bad" structures to see what the problem is, and attach the results here.</p> <p>Neil, James, since the <tt>obd_kobj</tt> and <tt>obd_ktype</tt> fields are recent additions and the largest fields in this area, it seems likely that they are the culprit here. Is there anything "special" about them that would require their alignment, or to avoid cacheline contention? Are they "hot" and referenced/refcounted continuously during object access?</p> <p>patch: <a href="https://review.whamcloud.com/41178" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41178</a> didn't help and it was still same slow down. let me try your "bisect" ideas to find if it shows good (or still bad) performance.</p> <p>Strange that the patch to reduce the structure to the previous size (-8 bytes) <b>and</b> the patch to move the new 8-byte field to the end didn't help performance <b>and</b> the patch to reduce the struct size -2000 bytes also didn't help. It is possible my "-8 byte" patch was too complex, and I moved too many fields around? I had tried to keep the related fields together.</p> <p>Do you have any idea where the performance is being lost? Is it high CPU usage or lock contention or something else?</p> <p>Attached are two frame graphes on good (reverted whole <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> patches) and bad (master) cases. <br/> I see huge amount of spin_locks in osc_queue_sync_pages() under ll_direct_IO() in bad case. There are also several spin_lock impacts in the functions under brw_interpret().</p> <p>It looks like the contended lock in <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/37143/37143_master.svg" title="master.svg attached to LU-14055">master.svg^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> is <tt>cl_loi_list_lock</tt> in both <tt>osc_queue_sync_pages()</tt> and <tt>brw_interpret()</tt>. The <tt>cl_loi_list_lock</tt> is in <tt>struct client_obd</tt>. What is strange is that this field is <b>before</b> the added structure, so I can't see how that would be affected by the new field. I'll have to check if there is something later in <tt>obd_device</tt> that is being accessed under that lock.</p> <p>&gt; Attached are two frame graphes on good (reverted whole <a href="https://jira.whamcloud.com/browse/LU-13344" title="Support for linux 5.6 clients" class="issue-link" data-issue-key="LU-13344"><del>LU-13344</del></a> patches) and bad (master) cases. </p> <p>How long did perf run for the two flame graphs? One has 5x the samples as the other. master-revert.svg shows 20% in swapper and 5% in open. Had ior completed startup when perf was run?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41162/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41162/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> lmv: reduce struct lmv_obd size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e11deeb1e6d114608eac4ee998d4cea22e30b0f5</p> <p>Ihara,</p> <p>Ignoring entirely the apparent memory layout issue, I have two suggestions that may reduce the pain on cl_loi_list_lock.</p> <p>This patch just landed to master:</p> <p><a href="https://review.whamcloud.com/38214" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38214</a></p> <p>And this one which has not yet landed:<br/> <a href="https://review.whamcloud.com/#/c/39482/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39482/</a></p> <p>If you're able to give these a try, that would be very interesting.  Both should cut the contention on cl_loi_list_lock when doing dio.  I'm not sure precisely what effect they'll have here, but it seems worth trying.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tappro" class="user-hover" rel="tappro">tappro</a> made this comment over on <a href="https://jira.whamcloud.com/browse/LU-14580" title="Lustre 2.12.6 performance regression" class="issue-link" data-issue-key="LU-14580">LU-14580</a>, and I wanted to bring it here:</p> <p> <br/> "I don't see problems with patch itself. Increment in <tt>osc_consume_write_grant()</tt> was removed because it is done by <tt>atomic_long_add_return()</tt> now outside that call and it is done in both places where it is called. But maybe the patch "<a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O" itself causes slowdown? Now grants are taken for Direct IO as well, so maybe that is related to not enough grants problem or similar. Are there any complains about grants on client during IOR run?"</p> <p>That patch definitely has performance implications.  Direct i/o will keep sending even when there are no grants - since it is already synchronous - but it significantly increases the load on the cl_loi_list_lock in some cases.  The patches I noted above are aimed at that.</p> <p>There's still very likely a memory layout issue here, but perhaps these will help...</p> <p>Ihara,</p> <p>FYI I had to rebase that second patch above:<br/> <a href="https://review.whamcloud.com/#/c/39482/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39482/</a></p> <p>It was not compatible with current master, should be fixed now.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a> can this issue now be considered fixed?</p> <p>My understanding is this issue is resolved, though we never fully understood it.  Let me knw if that's wrong and we can reopen or open a new ticket.</p> Related LU-8837 LU-13344 LU-14580 LU-13601 Development Rank 1|i01cun: Rank (Obsolete) 9223372036854775807 Severity