https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4139 Lustre <p>When a user goes over their softlimit there is a major performace hit.</p> <p>Testing showed a file copied in 3sec when under softlimit and 7 Min when over softlimit.</p> <p>Can be reproduced by just testing below and over softlimit.</p> <p>see trace for when the copy was slow.</p> LU-4139

Significant perforamce issue when user over soft quota limit

Bug Critical Resolved Fixed Niu Yawei Mahmoud Hanafi Wed, 23 Oct 2013 22:19:11 +0000 Wed, 4 Dec 2019 09:07:29 +0000 Tue, 3 Dec 2013 19:15:53 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 6 <p>Niu</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Mahmoud</p> <p>This is by design but not a bug.</p> <p>To acheive an accurate grace time management, we have to shrink the qunit size to the least qunit size (1K), then quota slave can only acquire 1K quota limit from master each time, that definitely hurts performance. For good performance, large qunit has to be kept when approaching softlimit, however, grace timer would not be triggered (or stopped) exactly on user's real disk usage, I think that would be really bad user experience.</p> <p>We think the performance when over softlimit (or approaching hardlimit) is less important comparing with an accurate grace timer (or an accurate -EDQUOT), so we choose to sacrifice the performance (in certain special case) for the accuracy. This is the reason of dynamic qunit introduced actually.</p> <p>Niu, when you write "1k qunit" is that really 1kB of data, or 1024 blocks? It doesn't make sense to only have 1kB of quota if the blocks are allocated in chunks of 4kB. </p> <p>3sec to 7 min is not a very good trade off. This behaviour makes softlimit useless for us. Is there a way we can tune how much min qunit is allocated if a user is over their softlimit.</p> <p>The qsd_internal.h showed the meaning of (qunit == 1024) as:<br/> 1MB or 1K inodes.</p> <blockquote> <p>The qsd_internal.h showed the meaning of (qunit == 1024) as:<br/> 1MB or 1K inodes.</p></blockquote> <p>Right, it is 1K blocks (1M).</p> <blockquote> <p>3sec to 7 min is not a very good trade off. This behaviour makes softlimit useless for us. Is there a way we can tune how much min qunit is allocated if a user is over their softlimit.</p></blockquote> <p>After over softlimit, client turns to sync write instead of writing to cache, and quota slave only acquires minimum quota limit which can just satisfy the incoming write each time, that means two RPC round-trip delay would be added to each write operation, but I don't think the performance gap will be so big (3sec to 7min), could you verify that all data was really flushed back (for the 3sec copy)? Or could you retry the test with direct io to see the difference? Thanks.</p> <p>Here is what my testing showed:</p> <p>Using IOR with directio 3 threads writing to single ost:<br/> Over Softlimit: 316MB/sec<br/> Under Softlimit: 356MB/sec</p> <p>Using IOR with buffered IO 3 threads writing to singe ost:<br/> Over Softlimit: 3.33MB/sec<br/> Under Softlimit: 2.91MB/sec</p> <p>using 'cp' to copy a 10G file<br/> Under Softlimit: 7.8sec<br/> Over softlimit: &gt;10Min. I didn't want to wait for it to finish</p> <p>What I noticed is when I was over my softlimit using cp all the I/O was 4KB RPCs. I was able to see this happen in the middle of my test as I would go over my softlimit the RPCs would drop from 1MB to 4KB. Also using IOR with buffered I/O all the RPCs where 4K. It seems that the smaller I/O sizes are the main issue.</p> <p>MY IOR COMMANDS<br/> Buffered IO<br/> mpiexec -n 3 /u/mhanafi/bin/IOR -a POSIX -C -Y -e -i 1 -w -t 1m -b 1g -v -O useExistingTestFile=1,keepFile=1,filePerProc=1,intraTestBarriers=1,testFile=ior,outlierThreshold=1</p> <p>Direct IO<br/> mpiexec -n 3 /u/mhanafi/bin/IOR -a POSIX -C -B -i 1 -w -t 1m -b 1g -v -O useExistingTestFile=1,keepFile=1,filePerProc=1,intraTestBarriers=1,testFile=ior,outlierThreshold=1</p> <p>Niu,<br/> I think it is unintentional that over softlimit IO is done in 4kB chunks, even if the qunit is getting 1MB chunks. Is it possible to avoid throttling the clients if there is a large gap between the soft and hard quota limit (i.e. treating over softlimit the same as under softlimit if there is still a large margin before the hardlimit)?</p> <blockquote> <p>What I noticed is when I was over my softlimit using cp all the I/O was 4KB RPCs. I was able to see this happen in the middle of my test as I would go over my softlimit the RPCs would drop from 1MB to 4KB. Also using IOR with buffered I/O all the RPCs where 4K. It seems that the smaller I/O sizes are the main issue.</p></blockquote> <p>For quota accuracy, when approaching (or over) quota hardlimit (or softlimit), client turns to sync write (see bug16642), and in consequence the PRC size will be page size, 4K (page can't be cached on client, it has to be synced out on write).</p> <blockquote> <p>I think it is unintentional that over softlimit IO is done in 4kB chunks, even if the qunit is getting 1MB chunks. Is it possible to avoid throttling the clients if there is a large gap between the soft and hard quota limit (i.e. treating over softlimit the same as under softlimit if there is still a large margin before the hardlimit)?</p></blockquote> <p>As I described above, page size (4KB) io is because of sync write on client. To avoid sync write on client after over softlimit, I think probably we can tweak the qunit size differently when over softlimit. I'll try to cooke a patch.</p> <p>While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.</p> <blockquote> <p>While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.</p></blockquote> <p>The new quota didn't change client protocol, so triggering sync write when approaching limit is same as before, please check the server code qsd_op_begin0():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> __u64 usage; lqe_read_lock(lqe); usage = lqe-&gt;lqe_usage; usage += lqe-&gt;lqe_pending_write; usage += lqe-&gt;lqe_waiting_write; usage += qqi-&gt;qqi_qsd-&gt;qsd_sync_threshold; <span class="code-comment">/* <span class="code-keyword">if</span> we should notify client to start sync write */</span> <span class="code-keyword">if</span> (usage &gt;= lqe-&gt;lqe_granted - lqe-&gt;lqe_pending_rel) *flags |= LQUOTA_OVER_FL(qqi-&gt;qqi_qtype); <span class="code-keyword">else</span> *flags &amp;= ~LQUOTA_OVER_FL(qqi-&gt;qqi_qtype); lqe_read_unlock(lqe); </pre> </div></div> <p>And the client code osc_queue_async_io() -&gt; osc_quota_chkdq().</p> <p>Lose some grace time accuracy to improve write performance when over softlimit: <a href="http://review.whamcloud.com/8078" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8078</a></p> <p>How does this patch help with the 4k io sizes? I think that is the real issue with the performance. </p> <blockquote> <p>How does this patch help with the 4k io sizes? I think that is the real issue with the performance.</p></blockquote> <p>4K io size is caused by over quota flag on client, with this patch, slave can acquire/pre-acquire little bit more spare limit each time when over softlimit, then over quota flag won't be set on client anymore.</p> <p>New benchmark number with the patch</p> <p>Direct I/O<br/> UnderSoftlimit: 383MB/sec<br/> OverSoftlimit: 359MB/sec</p> <p>Buffered I/O<br/> UnderSoftlimit:316MB.sec<br/> OverSoftlimit: 304MB/sec</p> <p>So it looks good! </p> <p>Patch <a href="http://review.whamcloud.com/8078" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8078</a> landed on master branch and was cherry-picked to Lustre b2_4 branch.</p> <p>Please close this one.</p> <p>ok Mahmoud</p> <p>backported to b2_5: <a href="http://review.whamcloud.com/8603" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8603</a></p> Related LU-7795 Development Rank 1|hzw6pr: Rank (Obsolete) 11230 Severity