[LU-12194] clients getting soft lockups on 2.10.7

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.10.7
• Environment: EL 7.4.1708
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Campbell Mcleay (Inactive)

Description

Getting occasional soft lockups on 2.10.7 clients

kernel: NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ptlrpcd_01_08:11711]

Issue Links

• is related to: LU-12667 Read doesn't perform well in complex NUMA configuration
• is related to: LU-11895 CPU lockup in LNetMDUnlink

Comments

[Yang Sheng]

Hi, Campbell,

I am testing the patch in our test cluster. Yes, I think it will be landed in next few months. You can setup it via cpu_pattern before that.

Thanks,
YangSheng

[Campbell Mcleay (Inactive)]

Hi YangSheng,

I didn't see a patch in the 2.10.7 -> 2.10.8 changelog that will set NUMA topology - you mentioned it may take some time to get this patched - do you think it may get done within the next few months? I'm just wondering whether to wait for the patches and upgrade.

Thanks,

Campbell

[Yang Sheng]

Hi, Campbell,

You can refer to document http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.libcfstuning. But we still haven't a detail standard for CPT configuration. Since it is really depend on situation.

Thanks,
Yangsheng

[Campbell Mcleay (Inactive)]

Hi YangSheng,

Are you able to confirm what the general rule is for partitioning?

Thanks,
Campbell

[Campbell Mcleay (Inactive)]

Hi YangSheng,

What is the general rule for setting cpu_npartitions - is it number of NUMA node cpus divided by no. of NUMA nodes?

Thanks,

Campbell

[Yang Sheng]

Hi， Campbell,

No, It will set cpt automatically. So we needn't set it by manually. We do it for UMA node. But looks like not on NUMA node.

Thanks,
Yangsheng

[Campbell Mcleay (Inactive)]

Thanks Yangsheng. So the proposed patch will be to modify ko2iblnd.conf?

[Yang Sheng]

Hi, Campbell,

I think you can apply this change to all of clients that might be impacted by this issue. I'll try to push a patch to make this change more easy. But i think it could take a long time. So can we close this one first?

BTW: you can back to your original version lustre to remove the debug patch.

Thanks,
Yangsheng

[Campbell Mcleay (Inactive)]

Hi YangSheng,

I'm not collecting spt_table_data at the moment, but I also haven't seen any soft lockups since the changes were made. So what next from here? Do I just add these options to all clients on 2.10.7? Or is there a patch imminent to prevent the issue with the default CPU topology?

Kind regards,

Campbell

[Yang Sheng]

Hi, Campbell,

Could you please tell me the status of site? Do you still collect spt_table data?

Thanks,
YangSheng