[LU-15095] lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.15.0
• Affects Version/s: Upstream
• Severity: 3
• Assignee: Vladimir Saveliev
• Reporter: Alex Zhuravlev

Description

I'm getting many messages like this:
lctl: error invoking upcall /usr/sbin/lctl set_param ..lbug_on_grant_miscount=1
in the logs.
this was introduced in bb5d81ea95 ("LU-14543 target: prevent overflowing of tgd->tgd_tot_granted")
I don't quite understand why this clearly debugging tunable needs to be persisten in the config logs?
IMO, the better would be to have node-wide non-persisten tunable like osd's track_declare_assert

Issue Links

• is related to: LU-14543 tgt_grant_discard(): avoid tgd->tgd_tot_granted overflowing

Comments

[Andreas Dilger]

Moving this patch to a module parameter is causing RHEL7.9 testing to fail 100% with:
https://testing.whamcloud.com/test_sets/8a2e1a74-c60f-4e33-bbfa-2c3a7efa7e13

ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
[console] Lustre: Lustre: Build Version: 2.14.56_68_g5914687
[console] ptlrpc: Unknown parameter `lbug_on_grant_miscount'
modprobe: ERROR: could not insert 'ptlrpc': Unknown symbol in module, or unknown parameter (see dmesg)

The build version is the same on the client and server, so it isn't a case of an old build being used on the client.

I think the problem is that this is a client-only build being tested, and the module parameter is only for the server, so it just doesn't exist on the el7.9 client. I think the test-framework needs to be changed to only set this parameter on the OSS and MDS and not the client nodes.

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/45521/
Subject: LU-15095 target: lbug_on_grant_miscount module parameter
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2c787065441ee60c6c163dc77851d0964f81a89c

[Gerrit Updater]

"Vladimir Saveliev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/45521
Subject: LU-15095 target: lbug_on_grant_miscount module parameter
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 91f914fe01c71280523c5fee3bf2d31db593c9e5

[Alex Zhuravlev]

Do you mean to make it as parameter of module ptlrpc?

yes

[Alex Zhuravlev]

in some cases this parameter is set (written) into the config few times.

to reproduce it should be enough to run llmount.sh locally:

[   30.040858] Lustre: Modifying parameter general.*.*.lbug_on_grant_miscount in log params
[   30.161330] LustreError: 5029:0:(obd_config.c:1326:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param *.*.lbug_on_grant_miscount=1: rc = -2; time 191us

[Vladimir Saveliev]

do we really need to save this to the log? why not use a variable like cfs_fail_loc or ldiskfs_track_declares_assert ?

Do you mean to make it as parameter of module ptlrpc?

int ldiskfs_track_declares_assert;
module_param(ldiskfs_track_declares_assert, int, 0644);

It sounds like a good idea, thanks.

[Vladimir Saveliev]

Do we want lbug_on_grant_miscount set across reboots?

Without that the parameter lbug_on_grant_miscount would get turned off in tests which include failover.
That is, it was made permanent intentionally.

I did not get "lctl: error invoking upcall" in my tests and will debug the issue.

 

[James A Simmons]

That is really good point. Do we want lbug_on_grant_miscount set across reboots?

[Alex Zhuravlev]

do we really need to save this to the log? why not use a variable like cfs_fail_loc or ldiskfs_track_declares_assert ?

[James A Simmons]

I think the simple solution is change ..lbug_on_grant_miscount=1 to [mdt|obdfilter].*.lbug_on_grant_miscount=1