[LU-6584] OSS hit LBUG and crash

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.7.0
• Labels: sdsc

Environment

[root@panda-oss-25-4 ~]# uname -a
Linux panda-oss-25-4.sdsc.edu 3.10.73-1.el6.elrepo.x86_64 #1 SMP Thu Mar 26 16:28:30 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@panda-oss-25-4 ~]# rpm -aq | grep lustre
lustre-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-osd-zfs-mount-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-iokit-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-source-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-osd-zfs-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-modules-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64
lustre-tests-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64

• Severity: 4
• Assignee: Mikhail Pershin
• Reporter: Haisong Cai (Inactive)

Description

No sign or indication, ie lustre-log or error messages, OSS unexpectantly crash (please see console image).

/var/log/messages is attached

Comments

[Peter Jones]

Will SDSC be able to try this patch out to confirm whether it fixes the issues that they have been experiencing?

[Mikhail Pershin]

It seems the reason of this issue is the int type overflow in lnb_rc. Instead of writing the (eof - file_offset) right into lnb_rc we have to check first it is not negative.

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16685
Subject: LU-6584 osd: prevent int type overflow in osd_read_prep()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 687338302147dad5b09b964b8615a3b3adb78a7d

[Rick Wagner (Inactive)]

Hi Andreas, since our last update to the code tree based on http://review.whamcloud.com/#/c/14926/ we've been stable. It's possible that we've pulled in a bugfix along with the debugging patch although I couldn't point to a specific one.

We are looking at ZFS 0.6.5 to get away from the unreleased version for ZFS we've had to run. I would probably do that along with another rebase to a later unpatched tag of Lustre, maybe once LU-4865 is included.

On a related note, I think this issue could be removed from the 2.8 blocker list, since we started with patched versions of Lustre and ZFS.

[Andreas Dilger]

Hi Rick, any news on this front? Have you looked into upgrading to ZFS 0.6.5 to get the native large block support? The patch http://review.whamcloud.com/15127 "LU-4865 zfs: grow block size by write pattern" should also help performance when dealing with files under 1MB in size.

[Rick Wagner (Inactive)]

We've scheduled a maintenance window for Sep. 8 to roll out this latest patch after testing.

Andreas, I'll consider changing the recordsize on some of the OSTs. The most likely scenario where we get solid information from this is if the LBUG is still hit on one of the OSSes with the changed setting. I am being a little cautious considering this since it will mean having a ZFS dataset with varying recordsizes. I don't believe the ZFS layer will care, but it's not something I've dealt with before.

[Zhenyu Xu]

http://review.whamcloud.com/#/c/14926/ has been updated to add more remote/local buffer check.

[Andreas Dilger]

Rick, the other possible avenue for debugging is to disable the 1MB blocksize tunable on one or more of your OST datasets, and see if this correlates to a reduction or elimination of the occurrence of this failure. This is one of the main deltas between your ZFS environment and other ZFS users, so this would allow us to isolate the memory corruption to the code handling 1MB blocksize.

[Andreas Dilger]

Bobijam,
can you please make a new patch that checks the contents of niobuf_remote when it is first accessed by the OST (tgt_brw_read() and tgt_brw_write()) to verify that the contents are sane, and print out all the values under D_BUFFS debugging. If the values are incorrect a CERROR() should be printed and an -EPROTO error returned to the client, and we can debug this problem as a network corruption.

This niobuf verification should be in a helper function that can also be called before the currently-failing LASSERT() checks are being handled (and elsewhere in the code if you think it is helpful), and those functions can return an EIO error to the caller rather than triggering the LASSERT. At that point the client should resend the BRW RPC due to brw_interpret()>osc_recoverable_error() and hopefully it will succeed on the second try.

While I don't think this is a proper solution, it will at least tell us if the corruption is happening on the client and/or on the network, or in memory on the OSS, and it will potentially allow debugging to continue without the high frequency of OSS failures.

[Rick Wagner (Inactive)]

Andreas, yes, we're using 1MB block sizes on the ZFS datasets that handle the OSTs.

[Dmitry Mishin (Inactive)]

We're using the f1512ee61e commit from master ZFS branch (large block support). It's later than 0.6.4.1, and I had problems running with latest master version.