https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15095 Lustre <p>I'm getting many messages like this:<br/> lctl: error invoking upcall /usr/sbin/lctl set_param <b>.</b>.lbug_on_grant_miscount=1<br/> in the logs.<br/> this was introduced in bb5d81ea95 ("<a href="https://jira.whamcloud.com/browse/LU-14543" title="tgt_grant_discard(): avoid tgd-&gt;tgd_tot_granted overflowing" class="issue-link" data-issue-key="LU-14543"><del>LU-14543</del></a> target: prevent overflowing of tgd-&gt;tgd_tot_granted")<br/> I don't quite understand why this clearly debugging tunable needs to be persisten in the config logs?<br/> IMO, the better would be to have node-wide non-persisten tunable like osd's track_declare_assert</p> LU-15095

lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1

Bug Critical Resolved Fixed Vladimir Saveliev Alex Zhuravlev Wed, 13 Oct 2021 06:54:06 +0000 Wed, 16 Mar 2022 23:46:29 +0000 Sat, 22 Jan 2022 08:41:24 +0000 Upstream Lustre 2.15.0 0 5 <p>yet another interesting side-effect of that patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == sanity test 901: don't leak a mgc lock on client umount ========================================================== 10:40:03 (1634121603) 192.168.121.177@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client tmp.hAgqpwOV43 /mnt/lustre (opts:) Starting client: tmp.hAgqpwOV43: -o user_xattr,flock tmp.hAgqpwOV43@tcp:/lustre /mnt/lustre sanity test_901: @@@@@@ FAIL: mgc lock leak (16 != 17) Trace dump: = ./../tests/test-framework.sh:6330:error() = sanity.sh:27444:test_901() = ./../tests/test-framework.sh:6634:run_one() = ./../tests/test-framework.sh:6681:run_one_logged() = ./../tests/test-framework.sh:6522:run_test() = sanity.sh:27449:main() </pre> </div></div> <p>I disabled this line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">do_node $(mgs_node) <span class="code-quote">"$LCTL set_param -P *.*.lbug_on_grant_miscount=1"</span></pre> </div></div> <p>and sanity/901 stops to fail</p> <p>Looking at the patch their is no lbug_on_grant_miscount on the MGS node. Only the MDS servers. Is lbug_on_grant_miscount meant for MGS servers or MDS servers?</p> <p>hmm, I think MGS was specified to make the parameter persistent, i.e. write it to the config logs for MDTs and OSTs (which do grants)</p> <p>Looking at it with fresh eyes and I see the -P now. Looking at the error I see a corner case missed. Normally when setting persistent values an udev event is sent out. In this case it doesn't and ends up with the fall back of using the upcall which does fail. The reason for this is that in process_param2_config() we use kset_find_obj() to find a top level kobj to use to send the event. In this case its '*' which doesn't exist. In this case we should just use the top kset kobject to send the event.</p> <p>I think the simple solution is change <b>.</b>.lbug_on_grant_miscount=1 to <span class="error">&#91;mdt|obdfilter&#93;</span>.*.lbug_on_grant_miscount=1</p> <p>do we really need to save this to the log? why not use a variable like cfs_fail_loc or ldiskfs_track_declares_assert ?</p> <p>That is really good point. Do we want lbug_on_grant_miscount set across reboots?</p> <blockquote> <p>Do we want lbug_on_grant_miscount set across reboots?</p></blockquote> <p>Without that the parameter lbug_on_grant_miscount would get turned off in tests which include failover.<br/> That is, it was made permanent intentionally.</p> <p>I did not get "lctl: error invoking upcall" in my tests and will debug the issue.</p> <p> </p> <blockquote> <p>do we really need to save this to the log? why not use a variable like cfs_fail_loc or ldiskfs_track_declares_assert ?</p></blockquote> <p>Do you mean to make it as parameter of module ptlrpc?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> ldiskfs_track_declares_assert; module_param(ldiskfs_track_declares_assert, <span class="code-object">int</span>, 0644); </pre> </div></div> <p>It sounds like a good idea, thanks.</p> <p>in some cases this parameter is set (written) into the config few times. </p> <p>to reproduce it should be enough to run llmount.sh locally:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 30.040858] Lustre: Modifying parameter general.*.*.lbug_on_grant_miscount in log params [ 30.161330] LustreError: 5029:0:(obd_config.c:1326:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1: rc = -2; time 191us </pre> </div></div> <blockquote><p>Do you mean to make it as parameter of module ptlrpc?</p></blockquote> <p>yes</p> <p>"Vladimir Saveliev &lt;vlaidimir.saveliev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45521" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45521</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15095" title="lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1" class="issue-link" data-issue-key="LU-15095"><del>LU-15095</del></a> target: lbug_on_grant_miscount module parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 91f914fe01c71280523c5fee3bf2d31db593c9e5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45521/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45521/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15095" title="lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1" class="issue-link" data-issue-key="LU-15095"><del>LU-15095</del></a> target: lbug_on_grant_miscount module parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2c787065441ee60c6c163dc77851d0964f81a89c</p> <p>Landed for 2.15</p> <p>Moving this patch to a module parameter is causing RHEL7.9 testing to fail 100% with:<br/> <a href="https://testing.whamcloud.com/test_sets/8a2e1a74-c60f-4e33-bbfa-2c3a7efa7e13" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8a2e1a74-c60f-4e33-bbfa-2c3a7efa7e13</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' [console] Lustre: Lustre: Build Version: 2.14.56_68_g5914687 [console] ptlrpc: Unknown parameter `lbug_on_grant_miscount' modprobe: ERROR: could not insert 'ptlrpc': Unknown symbol in module, or unknown parameter (see dmesg) </pre> </div></div> <p>The build version is the same on the client and server, so it isn't a case of an old build being used on the client. </p> <p>I think the problem is that this is a client-only build being tested, and the module parameter is only for the server, so it just doesn't exist on the el7.9 client. I think the test-framework needs to be changed to only set this parameter on the OSS and MDS and not the client nodes. </p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46185" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46185</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15095" title="lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1" class="issue-link" data-issue-key="LU-15095"><del>LU-15095</del></a> tests: skip lbug_on_grant_miscount on client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c168c309860dbf9745af1105ed3b236c0e2ce89c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46185/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46185/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15095" title="lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1" class="issue-link" data-issue-key="LU-15095"><del>LU-15095</del></a> tests: skip lbug_on_grant_miscount on client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 49e29f38343ce0389df0aecf308b0986de94c029</p> Related LU-14543 Development Rank 1|i0271z: Rank (Obsolete) 9223372036854775807 Severity