https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14580 Lustre <p>Between Lustre 2.12.5 and 2.12.6 releases, we have a major performance regression on large systems with multiple interfaces.</p> <p>On a 16 socket Superdome Flex with 8 EDR interfaces functioning as a client of a filesystem capable of &gt; 100 GB/s, using Lustre 2.12.5, we are measuring up to 50 GB/s. With Lustre 2.12.6, the same test is down to 13 GB/s.</p> <p>Reverting commit ID f92c7a161242c478658af09159a127bc21cba611 restores performance.</p> 16 Socket Superdome Flex w/8 EDR IB interfaces LU-14580

Lustre 2.12.6 performance regression

Bug Major Open Unresolved Neil Brown Stephen Champion performance Fri, 2 Apr 2021 15:40:46 +0000 Sun, 4 Jul 2021 14:52:42 +0000 Lustre 2.12.6 0 14 <p>FWIW, the test we are using is IOR w/ POSIX Direct IO:<br/> + mpirun sdf-1 240 /jet/home/champios/ior/bin/ior -i 3 -t 16m -b 13120m -s 1 -a POSIX --posix.odirect -e -F -w -k -E -D 30 -o /ocean/neocortex/tests/IOR-files/iorfile</p> <p>Thanks for the heads-up Steve!</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=schamp" class="user-hover" rel="schamp">schamp</a> instead of reverting the entire f92c7a1 change, would it be possible to just revert the following hunk and rerun your benchmark?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/include/obd.h b/lustre/include/obd.h index 2b5fb97..8545fd4 100644 --- a/lustre/include/obd.h +++ b/lustre/include/obd.h @@ -204,7 +204,6 @@ struct client_obd { /* the grant values are protected by loi_list_lock below */ unsigned long cl_dirty_pages; /* all _dirty_ in pages */ unsigned long cl_dirty_max_pages; /* allowed w/o rpc */ - unsigned long cl_dirty_transit; /* dirty synchronous */ unsigned long cl_avail_grant; /* bytes of credit for ost */ unsigned long cl_lost_grant; /* lost credits (trunc) */ /* grant consumed for dirty pages */ </pre> </div></div> <p>This looks to be a badly ported patch.</p> <p>Before the patch, osc_consume_write_grant() increments obd_dirty_pages.  After the patch it doesn't.</p> <p>There are two callers of osc_consume_write_grant().</p> <p>One of them, in osc_enter_cache_try() now increments obd_dirty_pages before calling osc_consume_write_grant()</p> <p>The other, in osc_queue_sync_pages(), hasn't been updated.  Presumably it needs an increment.  Though there is already an increment after the call.</p> <p>So the patch makes a behaviour change which wasn't intended.  I don't know what the correct behaviour is.</p> <p>@mpershin you did the backport: can you check it?</p> <p> </p> <p>Sure, I am checking that . These changes are combination of two patches actually: first ID is Ia047affc33fb9277e6c28a8f6d7d088c385b51a8 and next one is already referred f92c7a161242c478658af09159a127bc21cba611</p> <p>Their ports to 2.12 base were a bit different from master branch so I am checking if some functionality was broken. </p> <p>I don't see problems with patch itself. Increment in <tt>osc_consume_write_grant()</tt> was removed because it is done by <tt>atomic_long_add_return()</tt> now outside that call and it is done in both places where it is called. But maybe the patch "<a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O" itself causes slowdown? Now grants are taken for Direct IO as well, so maybe that is related to not enough grants problem or similar. Are there any complains about grants on client during IOR run?</p> <p>@John Hammond Good idea! I tried re-introducing the padding to no effect on x86_64. 128-bit cache line architectures may have different results.</p> <p>Did you get any updated results on this Steve?</p> <p>I'm pretty confident this and <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> are the same issue, so I'm snagging both of these.</p> <p>I'm going to leave this one open for now, but I think <a href="https://jira.whamcloud.com/browse/LU-14055" title="Write performance regression caused by an commit from LU-13344" class="issue-link" data-issue-key="LU-14055"><del>LU-14055</del></a> is the better place to continue the discussion.</p> Related LU-14055 LU-12820 Development Rank 1|i01r9r: Rank (Obsolete) 9223372036854775807 Severity