Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-13037] print tbf stats https://jira.whamcloud.com/browse/LU-13037 Lustre <p>We would like a way to dump current tbf stats. <br/> For example print all the UID being tracked and what are each UID bucket usage.</p> LU-13037

print tbf stats

Improvement Major Open Unresolved Li Xi Mahmoud Hanafi Sat, 30 Nov 2019 06:34:04 +0000 Thu, 1 Apr 2021 19:03:32 +0000 0 9 <p>Li xi</p> <p>How possible is this with the current design of TBF?</p> <p>Peter</p> <p>I am working on a patch that works similar like jobstat, but that is going to take a while.</p> <p>Two entries are added into each service: nrs_tbf_stats_reg and nrs_tbf_stats_hp. The first one for regular requests, and the second one for high priority requests. And TBF information of all client classifications will be dumped from each entry. Following is an example of the dumped information:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_stats_hp - key: _10.0.1.253@tcp_10_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: _10.0.1.253@tcp_4_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: dd.0_10.0.1.253@tcp_10_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: dd.0_10.0.1.253@tcp_4_0_0 refs: 9 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 # cat /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_stats_reg - key: _10.0.1.253@tcp_10_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: _10.0.1.253@tcp_4_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: dd.0_10.0.1.253@tcp_10_0_0 refs: 0 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 - key: dd.0_10.0.1.253@tcp_4_0_0 refs: 9 rule: default rpc_rate: 10000 ntoken: 2 token_depth: 3 </pre> </div></div> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13037" title="print tbf stats" class="issue-link" data-issue-key="LU-13037">LU-13037</a> nrs: dump stats of TBF clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 01af62b6a5305e9d4483dbbbb27aa003cd234099</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> Please feel free to let me know whether the dumped information is what you need.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> what do you think?</p> <p>Why do we get more than 1 stats for a specific cpt and queue_type. Here for uid 929411059 we see for cpt=0 and queue_type=reg we get 2 and for cpt=0 and queue_type=hp we get 2.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp13-srv1 /sys/kernel/debug/lustre/ost/OSS/ost_io # cat /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_stats| grep -A 4 929411059 - uid: 929411059 cpt: 0 queue_type: hp refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 0 queue_type: hp refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 0 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 0 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> -- </pre> </div></div> <p>Later for cpt=8 we only 1 for hp and 2 for reg.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - uid: 929411059 cpt: 8 queue_type: hp refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 8 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 8 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> I noticed this week that you are carrying this patch in your distribution. Sorry that we missed your question above. Have you had any other questions/comments about using this change? Do you think that we should proceed with landing it in its current form or is more work required?</p> <p>Sorry for late reply.</p> <blockquote><p>Why do we get more than 1 stats for a specific cpt and queue_type.</p></blockquote> <p>Understood comparing to this, a single stats would be easier to understand. However, this is determined by the internal design and implementation of request handling of Lustre which have good reasons too. And TBF has no other choice but to use and depend on it.</p> <p>Lustre seperate requests into two different types, regular (reg) requests and high priority (hp) requests. And handling of these two types of requests are seperated in order to make sure high priority requests won't be blocked by many regular requests.</p> <p>And Lustre divide CPUs into different patitions (cpt). Each patition handles RPC requests independently. </p> <p>Because of these existing mechansim, TBF have to set the RPC rate limitations seperately (maybe with the same values), and the stats are seperated for each cpt and request type.</p> <p>I understand that we have hp and reg. But We get 2 request for the same cpt and queue_type type</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - uid: 929411059 cpt: 0 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> -- - uid: 929411059 cpt: 0 queue_type: reg refs: 1 rule: <span class="code-keyword">default</span> -- </pre> </div></div> <p>Yes Peter we would like to get this landed.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> Sorry for the misunderstanding. I found a bug of the patch. Not sure whether that is the cause of the duplicated outputs. The patch will be refreshed soon anyway.</p> <p>Hi Mahmoud,<br/> I just updated the patch (<a href="https://review.whamcloud.com/#/c/36950/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36950/</a>), could you please try it again?</p> <p>Thanks,<br/> Qian</p> <p>I had compilation error in lustre-2.12.4 against CentOS 7.7 kernel :</p> <p>Making all in .<br/> /tmp/rpmbuild-lustre-jlan-UDyILlEP/BUILD/lustre-2.12.4/lustre/ptlrpc/nrs_tbf.c: In function 'nrs_tbf_stats_seq_show':<br/> /tmp/rpmbuild-lustre-jlan-UDyILlEP/BUILD/lustre-2.12.4/lustre/ptlrpc/nrs_tbf.c:3882:2: error: format '%u' expects argument of type 'unsigned int', but argument 3 has type '__u64' <span class="error">[-Werror=format=]</span><br/> seq_printf(p, "%u\n", cli->tc_rpc_rate);<br/> ^<br/> /tmp/rpmbuild-lustre-jlan-UDyILlEP/BUILD/lustre-2.12.4/lustre/ptlrpc/nrs_tbf.c: At top level:<br/> cc1: error: unrecognized command line option "-Wno-stringop-overflow" <span class="error">[-Werror]</span><br/> cc1: error: unrecognized command line option "-Wno-stringop-truncation" <span class="error">[-Werror]</span><br/> cc1: error: unrecognized command line option "-Wno-format-truncation" <span class="error">[-Werror]</span><br/> cc1: all warnings being treated as errors<br/> make<span class="error">[7]</span>: *** <span class="error">[/tmp/rpmbuild-lustre-jlan-UDyILlEP/BUILD/lustre-2.12.4/lustre/ptlrpc/nrs_tbf.o]</span> Error 1</p> <p>I do not know how I got those "unrecgonized command line option". Before applying this patch it was compiled fine.</p> <p>I build it based on the latest master branch.<br/> It seems that tc_rpc_rate is __u32, while in 2.12.4, it was __u64</p> <p>To make it pass the build for 2.12.4, you just need to modify it with:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> seq_printf(p, <span class="code-quote">"%llu\n"</span>, cli->tc_rpc_rate); </pre> </div></div> <p>Regards,<br/> Qian</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jaylan" class="user-hover" rel="jaylan">jaylan</a> any updates on testing this patch?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jaylan" class="user-hover" rel="jaylan">jaylan</a> you have not provided any feedback as to whether this patch meets your requirements. However, rumour has it that you are carrying this patch - does this mean that you can now provide us some feedback as to whether this patch is useful and should proceed with landing?</p> <p>We have been using the patch. But I think it needs addition work to be more useful. We will need to think about how it could improved. </p> Related Development Rank 1|i00q7z: Rank (Obsolete) 9223372036854775807