[LU-7867] OI scrubber causing performance issues

• Type: Question/Request
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.5.5
• Assignee: nasf (Inactive)
• Reporter: Dustin Leverman

Description

OI scrubber causing performance issues.

• There are orphaned objects that are causing the OI scrubber to start.
• Is there an online or offline way of running lfsck at the moment?

Issue Links

• is duplicated by: LU-7933 debugfs stat reports invalid EA entry in inode
• is related to: LU-7378 Error destroying object with RC115

Comments

[Matt Ezell]

For this file system, we have a mixture of clients - most are 2.5, but we also have some 1.8 and 2.7. I just ran some tests, and it seems that new files created under 2.5 or 2.7 are OK. But the file I created on the 1.8 client shows the strange PFID EA.

The 1.8 clients are going away soon, so no need to debug that client, but we do have a ton of files on disk with the strange PFID EA. My main concern is that a future lfsck run (under 2.8 servers, eventually) might mark these files as damaged and/or orphaned.

I'll run some tests with debugging later today and upload relevant logs. Thanks.

[nasf (Inactive)]

Matt,

Is the "test" an existing file (updated from old device) or new created one? If it is the latter case, how do you generate it? and would you please to retest test with -1 level debug collected on the client and related OST?

Your system is Lustre-2.5.5 based (both client and server), right? any additional patches on that?

(BTW, I have tried Lustre-2.5.5 in my local VM environment, but cannot reproduce your trouble - strange PFID EA. So I need more logs from you.)

Thanks!

[Andreas Dilger]

No, the parent doesn't look correct. It should exactly match the output from lfs fid2path, but it looks offset by 3 bytes for some reason. We can add the fix into the same patch, no need for a new ticket.

[Matt Ezell]

Andreas - is the parent fid being printed correctly? I ran a quick test on a non-corrupted OST:

# lfs getstripe test
test
lmm_stripe_count:   4
lmm_stripe_size:    1048576
lmm_pattern:        1
lmm_layout_gen:     0
lmm_stripe_offset:  123
        obdidx           objid           objid           group
           123          120572        0x1d6fc                0
           124          120522        0x1d6ca                0
           125          120541        0x1d6dd                0
           126          120607        0x1d71f                0
# lfs path2fid test
[0x200000410:0x2254:0x0]

# debugfs -c -R 'stat O/0/d28/120572' /dev/mapper/f1-ddn1a-l24
...
Extended attributes stored in inode body: 
  lma = "00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 fc d6 01 00 00 00 00 00 " (24)
  lma: fid=[0x100000000:0x1d6fc:0x0] compat=0 incompat=0
  fid = "54 22 00 10 04 00 00 02 00 00 00 00 00 00 00 00 " (16)
  fid: parent=[0x200000410002254:0x0:0x0] stripe=0

Should I open a separate ticket for that?

[Matt Ezell]

I used the build from Jenkins and it now correctly shows the EAs for /O/0/d8/54354408:

  lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 01 e4 20 03 00 00 00 00 " (24)
  lma: fid=[0x100000000:0x320e401:0x0] compat=8 incompat=0
  fid = "f8 95 00 22 29 03 00 02 00 00 00 00 02 00 00 00 " (16)
  fid: parent=[0x2000329220095f8:0x0:0x0] stripe=2

[Andreas Dilger]

Updated patch http://review.whamcloud.com/18999 should now resolve the problem introduced in the upstream code.

[Andreas Dilger]

Sorry, committed this comment too quickly.

Something strange is going on here. I'm just checking my local MDT and OST filesystems with debugfs 1.42.12.wc1 and it is printing the xattrs values properly. Checking with debugfs 1.42.13.wc4 appears to have the same problem, looking into it more closely.

[nasf (Inactive)]

Matt,

Would you please to apply above patch on your e2fsprogs that fixed the debugfs issue, and please run the patched debugfs on your system.

Thanks!

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18999
Subject: LU-7867 debugfs: locate EA value correctly
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4f64aa8135d0865fba88c3ee49981362ec0cb994

[Matt Ezell]

Note from my previous comment that I manually set the value length to 23 to avoid the 'invalid EA' message

I ran debugfs under gdb and manually set entry->e_value_size = 23.

We are using the latest version available from Intel: e2fsprogs-1.42.13.wc4-7.el6.x86_64.rpm

[root@f1-oss1d8 ~]# rpm -qi e2fsprogs
Name        : e2fsprogs                    Relocations: (not relocatable)
Version     : 1.42.13.wc4                       Vendor: (none)
Release     : 7.el6                         Build Date: Fri Dec 11 19:51:16 2015
Install Date: Wed Feb 24 17:44:21 2016         Build Host: onyx-8-sde1-el6-x8664.onyx.hpdd.intel.com
Group       : System Environment/Base       Source RPM: e2fsprogs-1.42.13.wc4-7.el6.src.rpm
Size        : 3170704                          License: GPLv2
Signature   : (none)
URL         : https://downloads.hpdd.intel.com/public/e2fsprogs/
Summary     : Utilities for managing ext2, ext3, and ext4 filesystems
Description :
The e2fsprogs package contains a number of utilities for creating,
checking, modifying, and correcting any inconsistencies in second,
third and fourth extended (ext2/ext3/ext4) filesystems. E2fsprogs
contains e2fsck (used to repair filesystem inconsistencies after an
unclean shutdown), mke2fs (used to initialize a partition to contain
an empty ext2 filesystem), debugfs (used to examine the internal
structure of a filesystem, to manually repair a corrupted
filesystem, or to create test cases for e2fsck), tune2fs (used to
modify filesystem parameters), and most of the other core ext2fs
filesystem utilities.

You should install the e2fsprogs package if you need to manage the
performance of an ext2, ext3, or ext4 filesystem.

[nasf (Inactive)]

The output of debugfs seems wrong, because the OSD got the self FID as [0x100000000:0x320e401:0x0], but the debugfs output is: lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 01 e4 20 03 00 00 00 " (23). So the value offset is wrong.

Which version e2fsprogs are you using? Is it possible to try newer e2fsprogs?
Before resolve the debugfs issue, please NOT remove these OST-objects manually, because we are not sure whether they are really the targets to be destroyed.