[LU-16690] kernel: obd_memory max: 1854996506, obd_memory current: 1854996506

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Severity: 2
• Assignee: Dongyang Li
• Reporter: Mahmoud Hanafi

Description

After upgrade to 2.15.2 server hung with errors

kernel: obd_memory max: 1854996506, obd_memory current: 1854996506

see attached logs for full set of longs.

Issue Links

• is related to: LU-16413 T10PI is broken for CentOS 8.x

Comments

[Peter Jones]

Ok so in that case let's assume that this is fixed unless future evidence comes to light that this is not the case

[Mahmoud Hanafi]

Peter,

I haven't been able to reproduce it on our test filesystem, I think it is too small. We may need to wait on a production filesystem during an extended dedicated. 

[Peter Jones]

Mahmoud

The LU-16413 fix has been merged to b2_15 and will be in the upcoming 2.15.3 release. Have you tested the effectiveness of this release?

Peter 

[Dongyang Li]

Andreas,
From the logs the kernel is 4.18.0-425.3.1.el8_lustre.x86_64.

I think we are seeing memory allocation issue is because of LU-16413, the patch was landed in master but not in 2.15.2.
Before the patch the bio-integrity kernel patch is broken for 4.18 kernels, it removed check in bio_integrity_prep() if the integrity payload is already allocated or not, and from osd we are calling bio_integrity_prep() twice, so we are leaking the integrity payload. Both are fixed in LU-16413.
I will port LU-16413 to b2_15.

[Andreas Dilger]

Mahmoud, which kernel is running on the servers for this system?

[Mahmoud Hanafi]

Lower the number threads help but eventualy the service hit the issue. But host that had ib_iser t10pi disable was find. So for now we have disabled t10pi and lowered the number of threads.

 

I will try to see if I can get a reproducer to help debug the issue.