[LU-8715] Regression from LU-8057 causes loading of fld.ko hung in 2.7.2

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Affects Version/s: Lustre 2.7.0
• Environment: lustre server nas-2.7.2-3nasS running in centos 6.7.
• Severity: 3
• Assignee: Amir Shehata (Inactive)
• Reporter: Jay Lan (Inactive)

Description

Since our nas-2.7.2-2nas rebased to b2_7_fe to nas-2.7.2-3nas, we found loading lustre module fld.ko hanged. Modprobe took 100% cpu time and could not be killed.

I identified the culprit of the problem using git bisect:
commit f23e22da88f07e95071ec76807aaa42ecd39e8ca
Author: Amitoj Kaur Chawla <[e-mail]>
Date: Thu Jun 16 23:12:03 2016 +0800

LU-8057 ko2iblnd: Replace sg++ with sg = sg_next(sg)

It was a b2_7_fe back port from the following one:
Lustre-commit: d226464acaacccd240da43dcc22372fbf8cb04a6
Lustre-change: http://review.whamcloud.com/19342

Issue Links

• is related to: LU-8057 o2iblnd driver is causing memory corruption due to improper handling of scatter list.

Comments

[Peter Jones]

ok thanks!

[Jay Lan (Inactive)]

This case can be closed. Thanks.

[James A Simmons]

Is this still a problem

 

[Doug Oucharek (Inactive)]

Yes, sorry about that. LU-8057 is what I was referring to.

[Jay Lan (Inactive)]

Doug,

You wrote in previous commet:
" If this needs to be fixed quickly, then removing LU-8085 from your build will be the best approach. As ORNL needs LU-8085, we cannot remove it from master."

Did you actually mean to write "LU-8057"? We do not have LU-8057 in our git repo...

[Doug Oucharek (Inactive)]

As best we can figure out, the change in LU-8057 has causes a little more memory to be used per connection. That is pushing your system over the edge.

As James has indicated, a proper fix will be to change how to allocate memory in LNet. That is going to take some time to get right as the potential to break all of LNet is pretty good.

I don't believe the fix for LU-8057 is critical for your setup. If this needs to be fixed quickly, then removing LU-8085 from your build will be the best approach. As ORNL needs LU-8085, we cannot remove it from master.

[Mahmoud Hanafi]

Any updates?

[James A Simmons]

Why not. The problem is the LIBCFS_ALLOC and FREE macros. Looking at the macros gave me a headache so no patch from me. I need to get into the right mental state to tackle it

[Doug Oucharek (Inactive)]

James, can we do that fix under this ticket?

[James A Simmons]

I know exactly what your problem is. We saw this problem in the lustre core some time ago and changed the OBD_ALLOC macros. The libcfs/LNet layer uses it own LIBCFS_ALLOC macros which means when the allocations are more than 2 pages in size they hit the vmalloc spinlock serialization issue. We need a fix for libcfs much like lustre had.