[LU-16224] rw_seq_cst_vs_drop_caches dies with SIGBUS

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Labels: hxr llnl
• Environment: lustre-2.15.1_5.llnl ; 4.18.0-372.26.1.1toss.t4.x86_64
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Olaf Faaland

Description

Running the reproducer from LU-14541 (rw_seq_cst_vs_drop_caches.c) fails about 50% of the time with Lustre 2.15.1 (both client and servers).

[root@mutt21:toss-5803-sigbus]# ./run_test /p/olaf{a,b}/faaland1/test/sigbustest
++ ./rw_seq_cst_vs_drop_caches /p/olafa/faaland1/test/sigbustest /p/olafb/faaland1/test/sigbustest
u = 60, v = { 60, 59 }
./run_test: line 11: 120055 Aborted                 (core dumped) ./rw_seq_cst_vs_drop_caches $1 $2
++ status=134
++ signum=6
++ case $signum in
++ echo FAIL with SIGBUS
FAIL with SIGBUS

Although it's not yet confirmed to be the same issue, we have two users reporting jobs dying with a bus error intermittently, when using Lustre for I/O, which is what prompted me to run this against Lustre 2.15.1.

Issue Links

• is related to: LU-16064 RPC from evicted client can corrupt data
• is related to: LU-16160 take ldlm lock when queue sync pages

Comments

[Patrick Farrell]

Olaf,

Please try https://review.whamcloud.com/c/fs/lustre-release/+/49647/ and https://review.whamcloud.com/c/fs/lustre-release/+/49653/ (both, applied in that order) the first is a similar but simpler fix for this issue, the second is a fix for a possible data inconsistency exposed (though not caused) by the first.  We will probably be taking this in preference to Bobi's patch.  (The Maloo failures on those are unrelated to the patches, I'm just waiting for review, etc, before retriggering testing.)

By the way, I've been able to reproduce this issue locally using the same sanity test Olaf referenced.  For me it's only reliably hittable in the presence of memory pressure, but that seems to be timing, since it shouldn't be a hard requirement for the bug to occur.  (Though might be involved in the real applications hitting the issue.)

[Olaf Faaland]

Peter,

My 2 users hitting what I suspect to be the same issue are running against older Lustre versions, 2.12 clients and 2.14 servers.

When Bobijam's backport allows rw_seq_cst_vs_drop_caches.c to succeed reliably, and passes the usual automated tests, then I'll pull it into our 2.15 branch and work on getting my users to run on our 2.15 machines. Right now the backport has a -1 from Maloo.

thanks

[Peter Jones]

Olaf

Do you have a reliable reproducer for this issue? Are able to test the effectiveness of the patch?

Peter

[Zhenyu Xu]

yes, I'd port it to b2_15 at https://review.whamcloud.com/c/fs/lustre-release/+/49553 

[Peter Jones]

Bobijam

Do I understand correctly you intend to port  https://review.whamcloud.com/c/fs/lustre-release/+/49534 to b2_15 and then ask LLNL to use that in their reproducer?

Peter

[Olaf Faaland]

Hi, do you have an update for this issue?  It is creating problems for at least two users.  Thanks

[Xing Huang]

2022-12-03: The b2_15 patch of LU-16160 is being worked on. The master patch of LU-16064 is being reviewed.

[Olaf Faaland]

Thank you for the update

[Xing Huang]

2022-11-12: The b2_15 patch of LU-16160 is being updated according to master one. The master patch of LU-16064 needs to be rebased.

[Zhenyu Xu]

A revised patch which is trying to address the review question is under review, when it's passed I'd update the backports.

[Olaf Faaland]

Hi Bobijam,

What is the status of the two backports? I saw that there was a review question for one, and the other seems to have a build issue.

thanks,
Olaf