https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16454 Lustre <p>I am trying to increase mdc.*.max_mod_rpcs_in_flight to grater than 8 but I get an error.</p> <p> </p> <p><tt># lctl set_param mdc.fs1-MDT0000-mdc-ffff902107a5e000.max_rpcs_in_flight=128</tt><br/> <tt>mdc.fs1-MDT0000-mdc-ffff902107a5e000.max_rpcs_in_flight=128</tt></p> <p><tt>#lctl set_param mdc.fs1-MDT0000-mdc-ffff902107a5e000.max_mod_rpcs_in_flight=127</tt><br/> <tt>error: set_param: setting /sys/fs/lustre/mdc/fs1-MDT0000-mdc-ffff902107a5e000/max_mod_rpcs_in_flight=127: Numerical result out of range</tt></p> <p><tt># lctl get_param version</tt><br/> <tt>version=2.15.1</tt></p> LU-16454

can't set max_mod_rpcs_in_flight > 8

Bug Critical Resolved Fixed Vitaliy Kuznetsov Mahmoud Hanafi Sat, 7 Jan 2023 22:57:01 +0000 Fri, 7 Jul 2023 02:54:35 +0000 Tue, 14 Feb 2023 06:14:33 +0000 Lustre 2.15.1 0 8 <p>Here is the error in debug logs</p> <p> </p> <p>00000020:00020000:0.0F:1673137999.151510:0:2742:0:(genops.c:2175:obd_set_max_mod_rpcs_in_flight()) fs1-MDT0000-mdc-ffff902107a5e000: can't set max_mod_rpcs_in_flight=9 higher than ocd_maxmodrpcs=8 returned by the server at connection</p> <p>I figured out the issue. It was module setting on the server. The documentation should be update to state that the server side module param should be updated first.</p> <p> </p> <p>Vitaliy</p> <p>We discussed this during the triage call today. Andreas has some suggestions of how to address this issue that he will share and then could you please follow up and implement?</p> <p>Thanks</p> <p>Peter</p> <p>Vitaliy, in my previous investigation of a similar issue in <a href="https://jira.whamcloud.com/browse/LU-14144" title="get and set Lustre module parameters via &quot;lctl get_param/set_param&quot;" class="issue-link" data-issue-key="LU-14144">LU-14144</a> I couldn't find any good reason in the code or commit history why <tt>max_mod_rpcs_per_client</tt> was specifically a module parameter on the server and not a regular sysfs parameter. There doesn't appear to be any runtime dependency on this value (i.e. it doesn't define a static number of slots for the per-client replies or anything), and the only thing it is used for is to pass the limit to the client. For the same reason, there also doesn't appear to be a particularly hard limitation why the client cannot change and exceed the server-provided parameter, except to avoid overloading the server with too many RPCs at once, but that may also be true of the current limit with a larger number of clients, no different than "<tt>max_rpcs_in_flight</tt>".</p> <p>It seems reasonable to add a per-MDT "<tt>max_mod_rpcs_in_flight</tt>" tunable parameter to <tt>lustre/mdt/mdt_lproc.c</tt> so that it can be set with "<tt>lctl set_param</tt>" at runtime, for example like <tt>async_commit_count</tt>. The global <tt>max_mod_rpcs_per_client</tt> parameter should be used as the initial value, and add "<tt>(deprecated)</tt>" to the module description in <tt>mdt_handler.c</tt>.</p> <p>Mahmoud, the console error message printed when the client limit is reached is "<tt>myth-MDT0000-mdc-ffff979380fc1800: can't set max_mod_rpcs_in_flight=32 higher than ocd_maxmodrpcs=8 returned by the server at connection</tt>" but I agree this isn't totally clear. Instead of reporting "<tt>ocd_maxmodrpcs</tt>" (which is an internal field name) it should report the new "<tt>mdt.myth-MDT0000.max_mod_rpcs_in_flight</tt>" parameter, which would steer the admin to the right location to change this value. However, in the current implementation it would still be necessary to unmount/remount (or at least force a client reconnection) if this parameter is changed. </p> <p>The main question is whether there is any value for the MDS to "limit" the value that can be set by the client (which is not done for <tt>max_rpcs_in_flight</tt> or most other parameters) or if the client should be able set this larger than the default value the MDT returned (maybe some upper limit like 4x or 8x the MDT limit)? That would allow something like "<tt>lctl set_param -P <b>.</b>.max_mod_rpcs_in_time</tt>" to affect both the clients and servers.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> Ok, I'll start working on a solution to this ticket. <br/> Thanks</p> <p>"Vitaliy Kuznetsov &lt;vkuznetsov@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49749" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49749</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16454" title="can&#39;t set max_mod_rpcs_in_flight &gt; 8" class="issue-link" data-issue-key="LU-16454"><del>LU-16454</del></a> component: Add a per-MDT "max_mod_rpcs_in_flight"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 23463fee16abd0821b95129b333c31f354cf8a94</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49749/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49749/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16454" title="can&#39;t set max_mod_rpcs_in_flight &gt; 8" class="issue-link" data-issue-key="LU-16454"><del>LU-16454</del></a> mdt: Add a per-MDT "max_mod_rpcs_in_flight"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f16c31ccd91d66caba69d3ceea6a61c1682df59e</p> <p>just got this locally:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 08:08:05 (1676362085) start mds service on tmp.8Qi9eBagDy Loading modules from /mnt/build/lustre/tests/.. detected 2 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: <span class="code-quote">'lbug_on_grant_miscount=1'</span> gss/krb5 is not supported quota/lquota options: <span class="code-quote">'hash_lqs_cur_bits=3'</span> Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 Started lustre-MDT0000 start mds service on tmp.8Qi9eBagDy Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 Started lustre-MDT0001 tmp.8Qi9eBagDy: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid tmp.8Qi9eBagDy: Reading test skip list from /tmp/ltest.config tmp.8Qi9eBagDy: EXCEPT=<span class="code-quote">"$EXCEPT 32 53 63 102 115 119 123F"</span> tmp.8Qi9eBagDy: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid tmp.8Qi9eBagDy: Reading test skip list from /tmp/ltest.config tmp.8Qi9eBagDy: EXCEPT=<span class="code-quote">"$EXCEPT 32 53 63 102 115 119 123F"</span> start ost1 service on tmp.8Qi9eBagDy Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 Started lustre-OST0000 tmp.8Qi9eBagDy: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid tmp.8Qi9eBagDy: Reading test skip list from /tmp/ltest.config tmp.8Qi9eBagDy: EXCEPT=<span class="code-quote">"$EXCEPT 32 53 63 102 115 119 123F"</span> mount lustre on /mnt/lustre..... Starting client: tmp.8Qi9eBagDy: -o user_xattr,flock tmp.8Qi9eBagDy@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff8a7866aad000.max_mod_rpcs_in_flight=1 max_mod_rpcs_in_flight set to 1 creating 2 files ... fail_loc=0x159 launch 0 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK fail_loc=0x159 launch 1 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK mdc.lustre-MDT0001-mdc-ffff8a7866aad000.max_mod_rpcs_in_flight=5 max_mod_rpcs_in_flight set to 5 creating 6 files ... fail_loc=0x159 launch 4 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK fail_loc=0x159 launch 5 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK mdt_max_mod_rpcs_in_flight is 8 8 umount lustre on /mnt/lustre..... Stopping client tmp.8Qi9eBagDy /mnt/lustre (opts:) error: set_param: setting /sys/fs/lustre/mdt/lustre-MDT0000/max_mod_rpcs_in_flight=16: Numerical result out of range mount lustre on /mnt/lustre..... Starting client: tmp.8Qi9eBagDy: -o user_xattr,flock tmp.8Qi9eBagDy@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff8a784a138000.max_rpcs_in_flight=17 error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff8a784a138000/max_mod_rpcs_in_flight=16: Numerical result out of range conf-sanity test_90b: @@@@@@ FAIL: Unable to set max_mod_rpcs_in_flight to 16 Trace dump: = ./../tests/test-framework.sh:6549:error() = conf-sanity.sh:7139:check_max_mod_rpcs_in_flight() = conf-sanity.sh:7291:test_90b() = ./../tests/test-framework.sh:6887:run_one() = ./../tests/test-framework.sh:6937:run_one_logged() = ./../tests/test-framework.sh:6773:run_test() = conf-sanity.sh:7299:main() </pre> </div></div> <p>Minor fix for limit in <a href="https://jira.whamcloud.com/browse/LU-16558" title="Сan&#39;t set max_mod_rpcs_in_flight &gt; 8 (Default value)" class="issue-link" data-issue-key="LU-16558"><del>LU-16558</del></a></p> Gantt End to End LU-16558 Related LU-14144 LU-13503 Development Rank 1|i039d3: Rank (Obsolete) 9223372036854775807 Severity