[LU-9054] sanity test_312: FAIL: blksz error: , expected: 4096 Created: 26/Jan/17  Updated: 21/Nov/22

Status: Reopened
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.10.0
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Jian Yu Assignee: Jinshan Xiong (Inactive)
Resolution: Unresolved Votes: 0
Labels: always_except, zfs
Environment:

FSTYPE=zfs

Issue Links:
Related
is related to LU-14692 deprecate use of OST FID SEQ 0 for MD... Resolved
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

sanity test 312 failed as follows in review-zfs-part-1 test session on master branch:

== sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 21:36:27 (1485380187)
CMD: trevis-36vm4 zfs get -p recordsize lustre-ost1/ost1
CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1
CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1 149 172 227 259 291 323 355 387 419 451 483 515 547 580 594 630 662 694 709 741 764 774 838 870 902 934 966 979 998 1027 1059 1091 1123 1155 1187 1219 1251 1291 1407 1439 1471 1503 4128 8482 12131 12164 12196 12228 12260
trevis-36vm4: zdb: dmu_bonus_hold(149) failed, errno 2
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.155851 s, 26.3 kB/s
CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1
/usr/lib64/lustre/tests/sanity.sh: line 15331: 16K5125121K51251216K1: value too great for base (error token is "16K5125121K51251216K1")
/usr/lib64/lustre/tests/sanity.sh: line 15364: [: -eq: unary operator expected
 sanity test_312: @@@@@@ FAIL: blksz error: , expected: 4096

Maloo reports:
https://testing.hpdd.intel.com/test_sets/0bd3c2a6-e35f-11e6-981b-5254006e85c2
https://testing.hpdd.intel.com/test_sets/2434ad12-e312-11e6-981b-5254006e85c2
https://testing.hpdd.intel.com/test_sets/7c8df6e6-c7d6-11e6-9ae2-5254006e85c2
https://testing.hpdd.intel.com/test_sets/ec2e0fb8-9676-11e6-bc10-5254006e85c2

 Comments   
Comment by Emoly Liu [ 19/Apr/17 ]

+1 on master:
https://testing.hpdd.intel.com/test_sets/8808cd20-249c-11e7-9073-5254006e85c2

Comment by Gerrit Updater [ 03/Aug/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: https://review.whamcloud.com/28343
Subject: LU-9054 tests: make zdb work with zpool dataset
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2dcb45d93787f5e7309395163aa76105775a1049

Comment by Jinshan Xiong (Inactive) [ 04/Aug/17 ]

I have filed a ticket at https://github.com/zfsonlinux/zfs/issues/6464 so that zfs upstream community can take a look.

Comment by Gerrit Updater [ 09/Aug/17 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/28343/
Subject: LU-9054 tests: disable test_312 due to zdb issue
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 05ebd8218108dccaa31c1dbc97cd4cc90dfd3f8d

Comment by Minh Diep [ 09/Aug/17 ]

Landed in 2.11

Comment by Gerrit Updater [ 11/Aug/17 ]

Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/28480
Subject: LU-9054 tests: disable test_312 due to zdb issue
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 63e5ef7d684616d4493933d1bcd710c24fcc447d

Comment by James Nunez (Inactive) [ 18/Aug/17 ]

Why did we resolved this ticket as fixed? The only patch that has landed disables sanity test 312, but the problem still remains. Shouldn't we keep this ticket open until we have a solution in place and can remove the test from the ALWAYS_EXCEPT list?

Comment by James Nunez (Inactive) [ 18/Aug/17 ]

Reopening this ticket until we support a version of ZFS that fixes this issue and we can run santiy test 312.

Comment by Gerrit Updater [ 18/Aug/17 ]

John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/28480/
Subject: LU-9054 tests: disable test_312 due to zdb issue
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 29b89d5689ea0a018aa05a6ea78bc90e5229ccff

Comment by James Nunez (Inactive) [ 27/Feb/19 ]

I'm opening this ticket again so we can remove the test from the ALWAYS_EXCEPT list.

Comment by Andreas Dilger [ 09/Nov/22 ]

+3 on Gerrit Janitor: https://testing.whamcloud.com/gerrit-janitor/25873/results.html

Comment by Andreas Dilger [ 09/Nov/22 ]

The ZDB+MMP issue looks like it was fixed in ZFS Issue #6464 and ZFS Issue #7797, which were included into ZFS 0.7.12, so that should no longer be the issue here. The test is failing with:

ustre/tests/sanity.sh: line 24502: 16K51251251251251216K16K16K16K16K16K16K16K16K16K16K16K16K16K51251216K1: value too great for base (error token is "16K51251251251251216K16K16K16K16K16K16K16K16K16K16K16K16K16K51251216K1")
/home/green/git/lustre-release/lustre/tests/sanity.sh: line 24531: [: -eq: unary operator expected
 sanity test_312: @@@@@@ FAIL: blksz error: , expected: 4096

so something about the zdb output is generating too many block size numbers (16K, 512 repeated). I'll push a debug patch shortly.

Comment by Gerrit Updater [ 09/Nov/22 ]

"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49086
Subject: LU-9054 tests: add debugging to sanity test_312
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c1cac847298c172f1953f24ab26ab5ea80c0c329

Comment by Andreas Dilger [ 21/Nov/22 ]

With the added debugging from patch 49086 it can be seen that the zdb command is dumping information about quite a number of different dnodes. I don't really know much about zdb, but it looks like it somehow gets hold of the d11/ directory dnode and is dumping all of the entries in that directory?

https://testing.whamcloud.com/gerrit-janitor/26584/testresults/sanity2-zfs-centos7_x86_64-centos7_x86_64-retry7/sanity.test_312.test_log.oleg369-client.log

== sanity test 312: make sure ZFS adjusts its block size by write pattern ======= 02:42:55 (1669016575)
Dataset lustre-ost1/ost1 [ZPL], ID 134, cr_txg 8, 128M, 11623 objects, rootbp DVA[0]=<0:60039800:200> DVA[1]=<0:70029400:200> 
  [L0 DMU objset] fletcher4 lz4 unencrypted LE contiguous unique double size=1000L/200P
  birth=2246L/2246P fill=11623 cksum=10b6d02f82:59e951cb7b4:102b004c9a51c:20ce609d50f9e1

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       625    1   128K     4K      0     512     4K    0.00  ZFS plain file
                                               312   bonus  System attributes
	dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/O/0/d11/3819
	uid     0
	gid     0
	atime	Mon Nov 21 01:39:50 2022
	mtime	Mon Nov 21 01:39:50 2022
	ctime	Mon Nov 21 01:39:50 2022
	crtime	Mon Nov 21 01:39:48 2022
	gen	298
	mode	107666
	size	0
	parent	672
	links	1
	pflags	800000000000
	rdev	0x0000000000000000
	SA xattrs: 160 bytes, 2 entries

		trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353\016\000\000\000\000\000\000
		trusted.fid = \007\004\000\000\002\000\000\000H\000\000\000\000\000\000\000\000\000\020\000\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
Indirect blocks:


    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       672    2   128K    16K    45K      1K    80K  100.00  ZFS directory
                                               152   bonus  System attributes
	dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 4
	path	/O/0/d11
	uid     0
	gid     0
	atime	Wed Dec 31 19:00:00 1969
	mtime	Wed Dec 31 19:00:00 1969
	ctime	Wed Dec 31 19:00:00 1969
	crtime	Mon Nov 21 01:25:48 2022
	gen	45
	mode	40755
	size	0
	parent	647
	links	0
	pflags	800000000000
	rdev	0x0000000000000000
	Fat ZAP stats:
		Pointer table:
			1024 elements
			zt_blk: 0
			zt_numblks: 0
			zt_shift: 10
			zt_blks_copied: 0
			zt_nextblk: 0
		ZAP entries: 353
		Leaf blocks: 4
		Total blocks: 5
		zap_block_type: 0x8000000000000001
		zap_magic: 0x2f52ab2ab
		zap_salt: 0x1ba92ce1
		Leafs with 2^n pointers:
			  8:      4 ****
		Blocks with n*5 entries:
			  9:      4 ****
		Blocks n/10 full:
			  4:      2 **
			  5:      2 **
		Entries with n chunks:
			  3:    353 ****************************************
		Buckets with n entries:
			  0:   1709 ****************************************
			  1:    325 ********
			  2:     14 *

		5195 = 2929 (type: Regular File)
		36395 = 11404 (type: Regular File)
		27531 = 1744 (type: Regular File)
                :
                [listing of many directory entries]
                :
		34475 = 9388 (type: Regular File)
		34955 = 10373 (type: Regular File)
		28395 = 2370 (type: Regular File)
Indirect blocks:
               0 L1  0:30037a00:400 20000L/400P F=5 B=2246/2246 cksum=952b158ae3:48309161f6e2:14863f0c3647dc:4689b429072f201
               0  L0 0:8002b400:200 4000L/200P F=1 B=2246/2246 cksum=12114b61b1:78931eb2743:198d94f2f4b5e:3ab0b9a86f3c18
            4000  L0 0:8001e800:1600 4000L/1600P F=1 B=2244/2244 cksum=1321a384bb3:34b1dafdd5595:5eb3345070e37f4:7e9ba1615b5ce75
            8000  L0 0:8002b600:1400 4000L/1400P F=1 B=2246/2246 cksum=12368be1ad5:31de62aa1d99b:537e7b8d91e37b8:832aa38a26f08b4d
            c000  L0 0:70c80e00:1600 4000L/1600P F=1 B=1555/1555 cksum=1247d01dc5a:3b38fbafe005a:6fb375255e34ced:926381771742ea8b
           10000  L0 0:70aa5a00:1400 4000L/1400P F=1 B=1236/1236 cksum=10cbedd48b9:2fe5bb1d703f5:514bc78624461fa:4adb094208689915

		segment [0000000000000000, 0000000000014000) size   80K

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       950    1   128K     4K      0     512     4K    0.00  ZFS plain file
                                               228   bonus  System attributes
	dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/O/0/d11/27403
	uid     0
	gid     0
	atime	Wed Dec 31 19:00:00 1969
	mtime	Wed Dec 31 19:00:00 1969
	ctime	Wed Dec 31 19:00:00 1969
	crtime	Mon Nov 21 02:20:34 2022
	gen	856
	mode	107666
	size	0
	parent	672
	links	1
	pflags	800000000000
	rdev	0x0000000000000000
	SA xattrs: 76 bytes, 1 entries

		trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\013k\000\000\000\000\000\000
Indirect blocks:


    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       982    1   128K     4K      0     512     4K    0.00  ZFS plain file
                                               228   bonus  System attributes
	dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/O/0/d11/27435
	uid     0
	gid     0
	atime	Wed Dec 31 19:00:00 1969
	mtime	Wed Dec 31 19:00:00 1969
	ctime	Wed Dec 31 19:00:00 1969
	crtime	Mon Nov 21 02:20:34 2022
	gen	856
	mode	107666
	size	0
	parent	672
	links	1
	pflags	800000000000
	rdev	0x0000000000000000
	SA xattrs: 76 bytes, 1 entries

		trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000+k\000\000\000\000\000\000
Indirect blocks:


    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
      1014    1   128K     4K      0     512     4K    0.00  ZFS plain file
                                               228   bonus  System attributes
	dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/O/0/d11/27883
	uid     0
	gid     0
	atime	Wed Dec 31 19:00:00 1969
	mtime	Wed Dec 31 19:00:00 1969
	ctime	Wed Dec 31 19:00:00 1969
	crtime	Mon Nov 21 02:20:35 2022
	gen	856
	mode	107666
	size	0
	parent	672
	links	1
	pflags	800000000000
	rdev	0x0000000000000000
	SA xattrs: 76 bytes, 1 entries

		trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353l\000\000\000\000\000\000
Indirect blocks:
:
[listing of dnodes continues]
:
    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
     38985    1   128K     4K      0     512     4K    0.00  ZFS plain file
                                               312   bonus  System attributes
	dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
	dnode maxblkid: 0
	path	/O/0/d11/44267
	uid     0
	gid     0
	atime	Mon Nov 21 02:27:17 2022
	mtime	Mon Nov 21 02:27:17 2022
	ctime	Mon Nov 21 02:27:17 2022
	crtime	Mon Nov 21 02:25:27 2022
	gen	1076
	mode	107666
	size	0
	parent	672
	links	1
	pflags	800000000000
	rdev	0x0000000000000000
	SA xattrs: 160 bytes, 2 entries

		trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353\254\000\000\000\000\000\000
		trusted.fid = B#\000\000\002\000\000\000\0267\000\000\000\000\000\000\000\000\020\000\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
Indirect blocks:



 sanity test_312: @@@@@@ FAIL: zfs_objid unset
Comment by Colin Faber [ 21/Nov/22 ]

bzzz  FYI ^

Generated at Sat Feb 10 02:22:51 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.