[#LU-9384] conf-sanity test 32b fails with 'list verification failed '

[LU-9384] conf-sanity test 32b fails with 'list verification failed ' Created: 21/Apr/17 Updated: 16/Jan/18 Resolved: 07/Jun/17
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.10.0, Lustre 2.11.0
Fix Version/s:	Lustre 2.10.0

Type:

Bug

Priority:

Critical

Reporter:

James Nunez (Inactive)

Assignee:

Yang Sheng

Resolution:

Fixed

Votes:

Labels:

None

Issue Links:

Related
is related to	~~LU-9349~~	PFL known issues tracking ticket	Resolved
is related to	~~LU-9207~~	Create new conf-sanity test_32 disk i...	Resolved

Severity:

Rank (Obsolete):

9223372036854775807

Description

After landing the patch for ~~LU-9207~~, conf-sanity test 32b is failing in a Lustre DNE configuration. In particular, the failures are for the upgrade from disk2_9-ldiskfs.tar.bz2.
From the test log, we can see

CMD: trevis-35vm7 cat /tmp/t32/list
/usr/lib64/lustre/tests
--- /tmp/t32/list.orig	2017-04-21 07:14:02.501123374 +0000
+++ /tmp/t32/list	2017-04-21 07:14:02.587123374 +0000
@@ -22,8 +22,8 @@
 total 0
 total 0
 total 0
-total 10276
-total 80
+total 10280
+total 96
 144115205272502295 -rw-r--r-- 1 60000 60000 10485760 1490901645 t32_qf_old
 144115205272502293 -rw-r--r-- 1 0 0  1160 1488511399 README
 144115205272502292 -rwxr-xr-x 1 0 0 14167 1456757151 powerman
 conf-sanity test_32b: @@@@@@ FAIL: list verification failed

So far, this looks like it is only impacting review-dne-part-1 test sessions.

conf-sanity test 32b started failing with the ‘list verification’ error on April 17, 2017. Logs for recent failures areat:
https://testing.hpdd.intel.com/test_sets/0821ecfe-2686-11e7-8920-5254006e85c2
https://testing.hpdd.intel.com/test_sets/e18c9b86-2688-11e7-b742-5254006e85c2
https://testing.hpdd.intel.com/test_sets/8fe37e7c-2696-11e7-8920-5254006e85c2

Comments

Comment by Sarah Liu [ 21/Apr/17 ]

It seems a lustre upgrade issue. I tried to mount the same disk2_9-ldisk image on both 2.9.0 system and 2.10 system and found

on 2.9, untar the image and mount as loop

[root@onyx-69 2.9]# mount -t lustre -o loop,writeconf,mgsnode=10.2.2.59 /tmp/2.9/mdt /mnt/mds1
[root@onyx-69 2.9]# mount -t lustre -o loop,writeconf,mgsnode=10.2.2.59 /tmp/2.9/ost /mnt/ost1/
[root@onyx-69 2.9]# mount -t lustre 10.2.2.59:/t32fs /mnt/lustre/
[root@onyx-69 2.9]# lfs df /mnt/lustre/
UUID                   1K-blocks        Used   Available Use% Mounted on
t32fs-MDT0000_UUID        125368        1732      114276   1% /mnt/lustre[MDT:0]
t32fs-OST0000_UUID        162888       19732      129156  13% /mnt/lustre[OST:0]

filesystem summary:       162888       19732      129156  13% /mnt/lustre    <---------------------different line

[root@onyx-69 2.9]# ll /mnt/lustre/init.d/
total 80   <-------------------------------------------------------different line
-rw-r--r-- 1 root root 15131 Sep 12  2016 functions
-rwxr-xr-x 1 root root  4608 Dec  7 16:41 lnet
-rwxr-xr-x 1 root root   868 Dec  7 16:41 lsvcgss
-rwxr-xr-x 1 root root 16783 Dec  7 16:41 lustre
-rwxr-xr-x 1 root root  2989 Sep 12  2016 netconsole
-rwxr-xr-x 1 root root  6643 Sep 12  2016 network
-rwxr-xr-x 1 root root 14167 Feb 29  2016 powerman
-rw-r--r-- 1 root root  1160 Mar  2 19:23 README

On 2.10, did the same thing but found the size is different, marked in arrow

[root@onyx-70 ~]# lfs df /mnt/lustre/
UUID                   1K-blocks        Used   Available Use% Mounted on
t32fs-MDT0000_UUID        125368        1892      114116   2% /mnt/lustre[MDT:0]
t32fs-OST0000_UUID        162888       19768      129120  13% /mnt/lustre[OST:0]

filesystem_summary:       162888       19768      129120  13% /mnt/lustre <-------different line

[root@onyx-70 ~]# ll /mnt/lustre/init.d/
total 96   <-------------different line
-rw-r--r-- 1 root root 15131 Sep 12  2016 functions
-rwxr-xr-x 1 root root  4608 Dec  7 16:41 lnet
-rwxr-xr-x 1 root root   868 Dec  7 16:41 lsvcgss
-rwxr-xr-x 1 root root 16783 Dec  7 16:41 lustre
-rwxr-xr-x 1 root root  2989 Sep 12  2016 netconsole
-rwxr-xr-x 1 root root  6643 Sep 12  2016 network
-rwxr-xr-x 1 root root 14167 Feb 29  2016 powerman
-rw-r--r-- 1 root root  1160 Mar  2 19:23 README

Comment by Andreas Dilger [ 21/Apr/17 ]

The values that are different are the allocated filesystem blocks on the OST (in lfs df output) and the allocated file blocks (in ls -l output). I don't think those differences are reasons to consider this test failed, since there may be any number of reasons for this difference (e.g. allocation of internal log files, slight changes to on-disk format, etc).

It would be interesting to find out why these differences exist to see if there is really a bug. Firstly, check "ls -ls" on the directory to see if the block allocation of any file has changed by 16KB, or if several files have changed by 4KB?

A good candidate may also be that the extra PFL xattr data that is being stored on the OST objects is (incorrectly) causing an external xattr block to be created for each file. That would cause a severe performance drop for ldiskfs filesystems upgraded to 2.10 with 256-byte OST inodes and would need to be fixed. We had similar problems many years ago when OSTs first started storing the "fid" xattr on 128-byte OST inodes, causing every file to get an external 4KB xattr block.

To check this, please do the following steps on the OST after the 2.10 upgrade:

lfs getstripe /mnt/lustre/init.d/* to get the OST object IDs of those files
run debugfs -c /tmp/2.9/ost/ on the device
use stat O/0/d(<objid> % 32)/<objid> to list each OST object to see if it has external xattr blocks (shown with a non-zero value for File ACL:)

The main suspect here is patch https://review.whamcloud.com/24882 LU-8998 pfl: enhance PFID EA for PFL which increases the size of the "fid" xattr on the OST objects, but was supposed to fit into the in-inode xattr space.

Comment by Gerrit Updater [ 22/Apr/17 ]

James Nunez (james.a.nunez@intel.com) uploaded a new patch: https://review.whamcloud.com/26787
Subject: ~~LU-9384~~ test: Add conf-sanity 32b,c to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1bdccfb92110c5148e75b9add4e42e8e89766214

Comment by Gerrit Updater [ 23/Apr/17 ]

Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/26789
Subject: ~~LU-9384~~ tests: skip 2.9 filesystem images on 2.10
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 74dc91c14513227b75958357030115b95d9a79bf

Comment by Gerrit Updater [ 23/Apr/17 ]

Andreas Dilger (andreas.dilger@intel.com) merged in patch https://review.whamcloud.com/26789/
Subject: ~~LU-9384~~ tests: skip 2.9 filesystem images on 2.10
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bfa524f46b1506919fb6dc5aa68679568a5f04a3

Comment by Andreas Dilger [ 24/Apr/17 ]

Patch landed has only skipped the failing test, ticket should not be closed yet.

Any patches to fix this issue need to remove the change from patch https://review.whamcloud.com/26789 so that the 2.9 images are used.

Comment by Sarah Liu [ 24/Apr/17 ]

Running ls -ls on both 2.9 and 2.10 got following. there are 4 files changed by 4KB:
on 2.9

[root@onyx-69 init.d]# ls -ls
total 80
16 -rw-r--r-- 1 root root 15131 Sep 12  2016 functions
 8 -rwxr-xr-x 1 root root  4608 Dec  7 16:41 lnet
 4 -rwxr-xr-x 1 root root   868 Dec  7 16:41 lsvcgss    <-----------
20 -rwxr-xr-x 1 root root 16783 Dec  7 16:41 lustre
 4 -rwxr-xr-x 1 root root  2989 Sep 12  2016 netconsole
 8 -rwxr-xr-x 1 root root  6643 Sep 12  2016 network <-----------
16 -rwxr-xr-x 1 root root 14167 Feb 29  2016 powerman  <-----------
 4 -rw-r--r-- 1 root root  1160 Mar  2 19:23 README  <--------------

on 2.10

[root@onyx-70 init.d]# ls -ls
total 96
16 -rw-r--r-- 1 root root 15131 Sep 12  2016 functions
 8 -rwxr-xr-x 1 root root  4608 Dec  7 16:41 lnet
 8 -rwxr-xr-x 1 root root   868 Dec  7 16:41 lsvcgss
20 -rwxr-xr-x 1 root root 16783 Dec  7 16:41 lustre
 4 -rwxr-xr-x 1 root root  2989 Sep 12  2016 netconsole
12 -rwxr-xr-x 1 root root  6643 Sep 12  2016 network
20 -rwxr-xr-x 1 root root 14167 Feb 29  2016 powerman
 8 -rw-r--r-- 1 root root  1160 Mar  2 19:23 README

Then on 2.10, for these 4 changed files, the File ACL shows non-zero value. For example for file lsvcgss:

/mnt/lustre/init.d/lsvcgss
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       1
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            15	          0xf	             0

debugfs:  stat O/0/d15/15
Inode: 154   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 2932758177    Version: 0x00000001:0000004c
User:     0   Group:     0   Project:     0   Size: 868
File ACL: 8398    Directory ACL: 0
Links: 1   Blockcount: 16
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x58dd5aa9:00000000 -- Thu Mar 30 12:21:13 2017
 atime: 0x58dd5aa9:00000000 -- Thu Mar 30 12:21:13 2017
 mtime: 0x5848ac2e:00000000 -- Wed Dec  7 16:41:18 2016
crtime: 0x58dd5a9d:a2bcc980 -- Thu Mar 30 12:21:01 2017
Size of extra inode fields: 32
Extended attributes stored in inode body: 
  lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 0f 00 00 00 00 00 00 00
 " (24)
  lma: fid=[0x100000000:0xf:0x0] compat=8 incompat=0
EXTENTS:
(0):39936
debugfs:

unchanged file shows 0, for example file functions

[root@onyx-70 ~]# lfs getstripe /mnt/lustre/init.d/*
/mnt/lustre/init.d/functions
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       1
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group
	     0	            12	          0xc	             0

debugfs:  stat O/0/d12/12
Inode: 151   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 2932758174    Version: 0x00000001:00000049
User:     0   Group:     0   Size: 15131
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 32
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x58dd5aa9:00000000 -- Thu Mar 30 12:21:13 2017
 atime: 0x58dd5aa9:00000000 -- Thu Mar 30 12:21:13 2017
 mtime: 0x57d687d9:00000000 -- Mon Sep 12 03:47:53 2016
crtime: 0x58dd5a9d:a2bcc980 -- Thu Mar 30 12:21:01 2017
Size of extra inode fields: 28
Extended attributes stored in inode body: 
  lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 0c 00 00 00 00 00 00 00
 " (24)
  lma: fid=[0x100000000:0xc:0x0] compat=8 incompat=0
  fid = "01 04 00 00 02 00 00 00 0f 00 00 00 00 00 00 00 " (16)
  fid: parent=[0x200000401:0xf:0x0] stripe=0
EXTENTS:
(0-3):38912-38915
debugfs:

Comment by Andreas Dilger [ 24/Apr/17 ]

Hmm, it looks like the core inode size may have also changed in this release due to project quota. The larger inodes have 32 bytes of extended inode attributes:

Size of extra inode fields: 32
Extended attributes stored in inode body: 
  lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 0f 00 00 00 00 00 00 00
 " (24)
  lma: fid=[0x100000000:0xf:0x0] compat=8 incompat=0

vs. only 28 bytes for the smaller inodes, and the fid xattr still fits in the inode:

Size of extra inode fields: 28
Extended attributes stored in inode body: 
  lma = "08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 0c 00 00 00 00 00 00 00
 " (24)
  lma: fid=[0x100000000:0xc:0x0] compat=8 incompat=0
  fid = "01 04 00 00 02 00 00 00 0f 00 00 00 00 00 00 00 " (16)
  fid: parent=[0x200000401:0xf:0x0] stripe=0

Comment by Andreas Dilger [ 24/Apr/17 ]

struct ext4_inode {
        :
        :
/*80*/  __le16  i_extra_isize;
        __le16  i_checksum_hi;  /* crc32c(uuid+inum+inode) BE */
/*84*/  __le32  i_ctime_extra;  /* extra Change time      (nsec << 2 | epoch) */
/*88*/  __le32  i_mtime_extra;  /* extra Modification time(nsec << 2 | epoch) */
/*8c*/ __le32  i_atime_extra;  /* extra Access time      (nsec << 2 | epoch) */
/*90*/  __le32  i_crtime;       /* File Creation time */
/*94*/  __le32  i_crtime_extra; /* extra FileCreationtime (nsec << 2 | epoch) */
/*98*/  __le32  i_version_hi;   /* high 32 bits for 64-bit version */
/*9c*/  __le32  i_projid;       /* Project ID */
}; /* = 0xa0 = 160 bytes */

struct ext4_xattr_ibody_header {
        __le32  h_magic;        /* magic number for identification */
};

struct ext4_xattr_entry {
/*00*/  __u8    e_name_len;     /* length of name */
        __u8    e_name_index;   /* attribute name index */
        __le16  e_value_offs;   /* offset in disk block of value */
/*04*/  __le32  e_value_inum;   /* inode in which the value is stored */
/*08*/  __le32  e_value_size;   /* size of attribute value */
/*0c*/  __le32  e_hash;         /* hash value of name and value */
/*10*/  char    e_name[0];      /* attribute name */
}; /* = 0x14 = 20 bytes including 3-byte xattr name */

struct lustre_mdt_attrs {
        /**     
         * Bitfield for supported data in this structure. From enum lma_compat.
         * lma_self_fid and lma_flags are always available.
         */
/*00*/  __u32   lma_compat;
        /**
         * Per-file incompat feature list. Lustre version should support all
         * flags set in this field. The supported feature mask is available in
         * LMA_INCOMPAT_SUPP.
         */
/*04*/ __u32   lma_incompat;   
        /** FID of this inode */
/*08*/  struct lu_fid  lma_self_fid;
}; /* = 0x18 = 24 bytes */

struct lustre_ost_attrs {       
        /* Use lustre_mdt_attrs directly for now, need a common header
         * structure if want to change lustre_mdt_attrs in future. */
/*00*/  struct lustre_mdt_attrs loa_lma;
        
        /* Below five elements are for OST-object's PFID EA, the
         * lma_parent_fid::f_ver is composed of the stripe_count (high 16 bits)
         * and the stripe_index (low 16 bits), the size should not exceed
         * 5 * sizeof(__u64)) to be accessable by old Lustre. If the flag
         * LMAC_STRIPE_INFO is set, then loa_parent_fid and loa_stripe_size
         * are valid; if the flag LMAC_COMP_INFO is set, then the next three
         * loa_comp_* elements are valid. */
/*18*/  struct lu_fid   loa_parent_fid;
/*28*/  __u32           loa_stripe_size;
/*2c*/  __u32           loa_comp_id;
/*30*/  __u64           loa_comp_start;
/*38*/  __u64           loa_comp_end;
}; /* = 0x40 = 64 bytes */

This means we have 256 - 160 - 4 = 92 bytes for xattrs, 20 bytes for the loa ext4_xattr_entry, and 64 bytes for lustre_ost_attrs, and 4 bytes for the NULL separator between the last ext4_xattr_entry and the first value. That should be enough to fit with the packed "lma+fid in a single xattr" format. If the "fid" xattr is using the larger struct filter_fid that contains ff_layout then it will not fit:

struct filter_fid {
       struct lu_fid           ff_parent;
       struct ost_layout       ff_layout;
} __attribute__((packed));