[LU-8715] Regression from LU-8057 causes loading of fld.ko hung in 2.7.2

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Affects Version/s: Lustre 2.7.0
• Environment: lustre server nas-2.7.2-3nasS running in centos 6.7.
• Severity: 3
• Assignee: Amir Shehata (Inactive)
• Reporter: Jay Lan (Inactive)

Description

Since our nas-2.7.2-2nas rebased to b2_7_fe to nas-2.7.2-3nas, we found loading lustre module fld.ko hanged. Modprobe took 100% cpu time and could not be killed.

I identified the culprit of the problem using git bisect:
commit f23e22da88f07e95071ec76807aaa42ecd39e8ca
Author: Amitoj Kaur Chawla <[e-mail]>
Date: Thu Jun 16 23:12:03 2016 +0800

LU-8057 ko2iblnd: Replace sg++ with sg = sg_next(sg)

It was a b2_7_fe back port from the following one:
Lustre-commit: d226464acaacccd240da43dcc22372fbf8cb04a6
Lustre-change: http://review.whamcloud.com/19342

Issue Links

• is related to: LU-8057 o2iblnd driver is causing memory corruption due to improper handling of scatter list.

Comments

[Mahmoud Hanafi]

we have >12,000 clients. We do see some servers consume all the credits.

[Jay Lan (Inactive)]

@Bruno Faccini: Yes, I can reproduce the problem on our freshly rebooted lustre servers by doing 'modprobe fld.'

[Joseph Gmitter (Inactive)]

Hi Doug,

Can you please have a look into the issue since it relates to the LU-8057 change?

Thanks.
Joe

[James A Simmons]

The fix is correct and it fixes a real bug. What this change did is exposed another problem in the ko2iblnd driver. I have to ask is your system really consuming all those credits? I don't think the IB driver queue pair depth is big enough to handle all those credits.

[Mahmoud Hanafi]

Module load time before was about 2-5mins, because we have large ntx values.
(options ko2iblnd ntx=125536 credits=62768 fmr_pool_size=31385)
But after the patch it takes >20mins

[Bruno Faccini (Inactive)]

Well, both the failure and suspected cause look surprising.
Do you mean that the fld.ko module load simply hangs on a fresh system when running "modprobre lustre"?