[LU-7867] OI scrubber causing performance issues

• Type: Question/Request
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.5.5
• Assignee: nasf (Inactive)
• Reporter: Dustin Leverman

Description

OI scrubber causing performance issues.

• There are orphaned objects that are causing the OI scrubber to start.
• Is there an online or offline way of running lfsck at the moment?

Issue Links

• is duplicated by: LU-7933 debugfs stat reports invalid EA entry in inode
• is related to: LU-7378 Error destroying object with RC115

Comments

[nasf (Inactive)]

Related patches have been landed.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/19493/
Subject: LU-7867 e2fsprogs: update build version to 1.42.13.wc5
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 00c372887fff6d4a9baae075c3fc1523c47d8ad4

[nasf (Inactive)]

Thanks Dustin for the explanation. Then we will close the ticket after landing Andreas' latest patch.

[Dustin Leverman]

We are in a holding pattern on this until the customer allows us to take an outage. It will probably be another 5-6 weeks before we get the green light. I think it is okay to close this ticket.

Our plan is to run the e2fsck on the LUN, if that doesn't fix the issue we will wipe out the 2 OST objects by hand and will lfsck when we update to lustre-2.8 servers.

Thanks,
Dustin

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19493
Subject: LU-7867 e2fsprogs: update build version to 1.42.13.wc5
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: fae5f9fa7b3bb6bf134f0560b163276f9f6187a3

[nasf (Inactive)]

Any further information about the issue? Have we ever run e2fsck as discussed above?

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/18999/
Subject: LU-7867 debugfs: fix check for out-of-bound xattr value
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 595b51179eaafdc1c50ab8348cb83d9429aa2bfa

[Andreas Dilger]

I suspect that e2fsck will just consider these files as hard linked, so you will find them by looking for regular files under O/0/ with nlink=2.

[Matt Ezell]

I think the strange PFID EA is coming from fid_flatten. It takes the FID and does

ino = (seq << 24) + ((seq >> 24) & 0xffffff0000ULL) + fid_oid(fid);

to make the 128-bit FID fit in a 64-bit inode. I didn't specifically see this in the log, but I believe this is what the client is setting as PFID. Since the 1.8 clients are going away, it's not worth looking into it further. We'll make sure to test LFSCK in 2.8 against objects created under 1.8 before upgrading the server.

So, back to the original issue at hand. I guess we need to take a downtime to run e2fsck. There could be two outcomes:

• e2fsck identifies these objects as damaged and repairs or unlinks them
• It doesn't notice the issue and runs cleanly

If situation #2 happens, should we manually unlink them? Then, how do we ensure there aren't more damaged objects on this OST?

[nasf (Inactive)]

It is the client to send the PFID information to the OST when write/setattr. According to your description, the 1.8 client may sent "ostid" formatted PFID to the OST, then caused the strange PFID EA. Usually, as long as the OST-object is NOT real orphan, then after upgrading your system to Lustre-2.6 or newer, the layout LFSCK will handle it as unmatched MDT-OST objects pairs, and will correct the PFID EA with the right MDT-object's FID.