https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9054 Lustre <p>sanity test 312 failed as follows in review-zfs-part-1 test session on master branch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 21:36:27 (1485380187) CMD: trevis-36vm4 zfs get -p recordsize lustre-ost1/ost1 CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1 CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1 149 172 227 259 291 323 355 387 419 451 483 515 547 580 594 630 662 694 709 741 764 774 838 870 902 934 966 979 998 1027 1059 1091 1123 1155 1187 1219 1251 1291 1407 1439 1471 1503 4128 8482 12131 12164 12196 12228 12260 trevis-36vm4: zdb: dmu_bonus_hold(149) failed, errno 2 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.155851 s, 26.3 kB/s CMD: trevis-36vm4 zdb -e -p /dev/lvm-Role_OSS -dddd lustre-ost1/ost1 /usr/lib64/lustre/tests/sanity.sh: line 15331: 16K5125121K51251216K1: value too great for base (error token is "16K5125121K51251216K1") /usr/lib64/lustre/tests/sanity.sh: line 15364: [: -eq: unary operator expected sanity test_312: @@@@@@ FAIL: blksz error: , expected: 4096 </pre> </div></div> <p>Maloo reports:<br/> <a href="https://testing.hpdd.intel.com/test_sets/0bd3c2a6-e35f-11e6-981b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0bd3c2a6-e35f-11e6-981b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2434ad12-e312-11e6-981b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2434ad12-e312-11e6-981b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/7c8df6e6-c7d6-11e6-9ae2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7c8df6e6-c7d6-11e6-9ae2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/ec2e0fb8-9676-11e6-bc10-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ec2e0fb8-9676-11e6-bc10-5254006e85c2</a></p> FSTYPE=zfs LU-9054

sanity test_312: FAIL: blksz error: , expected: 4096

Bug Minor Reopened Unresolved Jinshan Xiong Jian Yu always_except zfs Thu, 26 Jan 2017 03:01:08 +0000 Mon, 21 Nov 2022 18:33:36 +0000 Lustre 2.10.0 0 5 <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8808cd20-249c-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8808cd20-249c-11e7-9073-5254006e85c2</a></p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28343" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28343</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9054" title="sanity test_312: FAIL: blksz error: , expected: 4096 " class="issue-link" data-issue-key="LU-9054">LU-9054</a> tests: make zdb work with zpool dataset<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2dcb45d93787f5e7309395163aa76105775a1049</p> <p>I have filed a ticket at <a href="https://github.com/zfsonlinux/zfs/issues/6464" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/issues/6464</a> so that zfs upstream community can take a look.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28343/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28343/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9054" title="sanity test_312: FAIL: blksz error: , expected: 4096 " class="issue-link" data-issue-key="LU-9054">LU-9054</a> tests: disable test_312 due to zdb issue<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 05ebd8218108dccaa31c1dbc97cd4cc90dfd3f8d</p> <p>Landed in 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28480" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28480</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9054" title="sanity test_312: FAIL: blksz error: , expected: 4096 " class="issue-link" data-issue-key="LU-9054">LU-9054</a> tests: disable test_312 due to zdb issue<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 63e5ef7d684616d4493933d1bcd710c24fcc447d</p> <p>Why did we resolved this ticket as fixed? The only patch that has landed disables sanity test 312, but the problem still remains. Shouldn't we keep this ticket open until we have a solution in place and can remove the test from the ALWAYS_EXCEPT list?</p> <p>Reopening this ticket until we support a version of ZFS that fixes this issue and we can run santiy test 312.</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28480/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28480/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9054" title="sanity test_312: FAIL: blksz error: , expected: 4096 " class="issue-link" data-issue-key="LU-9054">LU-9054</a> tests: disable test_312 due to zdb issue<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 29b89d5689ea0a018aa05a6ea78bc90e5229ccff</p> <p>I'm opening this ticket again so we can remove the test from the ALWAYS_EXCEPT list.</p> <p>+3 on Gerrit Janitor: <a href="https://testing.whamcloud.com/gerrit-janitor/25873/results.html" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/gerrit-janitor/25873/results.html</a></p> <p>The ZDB+MMP issue looks like it was fixed in <a href="https://github.com/openzfs/zfs/issues/6464" class="external-link" target="_blank" rel="nofollow noopener">ZFS Issue #6464</a> and <a href="https://github.com/openzfs/zfs/issues/7797" class="external-link" target="_blank" rel="nofollow noopener">ZFS Issue #7797</a>, which were included into ZFS 0.7.12, so that should no longer be the issue here. The test is failing with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ustre/tests/sanity.sh: line 24502: 16K51251251251251216K16K16K16K16K16K16K16K16K16K16K16K16K16K51251216K1: value too great for base (error token is "16K51251251251251216K16K16K16K16K16K16K16K16K16K16K16K16K16K51251216K1") /home/green/git/lustre-release/lustre/tests/sanity.sh: line 24531: [: -eq: unary operator expected sanity test_312: @@@@@@ FAIL: blksz error: , expected: 4096 </pre> </div></div> <p>so something about the <tt>zdb</tt> output is generating too many block size numbers (16K, 512 repeated). I'll push a debug patch shortly.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49086" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49086</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9054" title="sanity test_312: FAIL: blksz error: , expected: 4096 " class="issue-link" data-issue-key="LU-9054">LU-9054</a> tests: add debugging to sanity test_312<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c1cac847298c172f1953f24ab26ab5ea80c0c329</p> <p>With the added debugging from patch 49086 it can be seen that the zdb command is dumping information about quite a number of different dnodes. I don't really know much about zdb, but it looks like it somehow gets hold of the <tt>d11/</tt> directory dnode and is dumping all of the entries in that directory?</p> <p><a href="https://testing.whamcloud.com/gerrit-janitor/26584/testresults/sanity2-zfs-centos7_x86_64-centos7_x86_64-retry7/sanity.test_312.test_log.oleg369-client.log" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/gerrit-janitor/26584/testresults/sanity2-zfs-centos7_x86_64-centos7_x86_64-retry7/sanity.test_312.test_log.oleg369-client.log</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 312: make sure ZFS adjusts its block size by write pattern ======= 02:42:55 (1669016575) Dataset lustre-ost1/ost1 [ZPL], ID 134, cr_txg 8, 128M, 11623 objects, rootbp DVA[0]=&lt;0:60039800:200&gt; DVA[1]=&lt;0:70029400:200&gt; [L0 DMU objset] fletcher4 lz4 unencrypted LE contiguous unique double size=1000L/200P birth=2246L/2246P fill=11623 cksum=10b6d02f82:59e951cb7b4:102b004c9a51c:20ce609d50f9e1 Object lvl iblk dblk dsize dnsize lsize %full type 625 1 128K 4K 0 512 4K 0.00 ZFS plain file 312 bonus System attributes dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 0 path /O/0/d11/3819 uid 0 gid 0 atime Mon Nov 21 01:39:50 2022 mtime Mon Nov 21 01:39:50 2022 ctime Mon Nov 21 01:39:50 2022 crtime Mon Nov 21 01:39:48 2022 gen 298 mode 107666 size 0 parent 672 links 1 pflags 800000000000 rdev 0x0000000000000000 SA xattrs: 160 bytes, 2 entries trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353\016\000\000\000\000\000\000 trusted.fid = \007\004\000\000\002\000\000\000H\000\000\000\000\000\000\000\000\000\020\000\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000 Indirect blocks: Object lvl iblk dblk dsize dnsize lsize %full type 672 2 128K 16K 45K 1K 80K 100.00 ZFS directory 152 bonus System attributes dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 4 path /O/0/d11 uid 0 gid 0 atime Wed Dec 31 19:00:00 1969 mtime Wed Dec 31 19:00:00 1969 ctime Wed Dec 31 19:00:00 1969 crtime Mon Nov 21 01:25:48 2022 gen 45 mode 40755 size 0 parent 647 links 0 pflags 800000000000 rdev 0x0000000000000000 Fat ZAP stats: Pointer table: 1024 elements zt_blk: 0 zt_numblks: 0 zt_shift: 10 zt_blks_copied: 0 zt_nextblk: 0 ZAP entries: 353 Leaf blocks: 4 Total blocks: 5 zap_block_type: 0x8000000000000001 zap_magic: 0x2f52ab2ab zap_salt: 0x1ba92ce1 Leafs with 2^n pointers: 8: 4 **** Blocks with n*5 entries: 9: 4 **** Blocks n/10 full: 4: 2 ** 5: 2 ** Entries with n chunks: 3: 353 **************************************** Buckets with n entries: 0: 1709 **************************************** 1: 325 ******** 2: 14 * 5195 = 2929 (type: Regular File) 36395 = 11404 (type: Regular File) 27531 = 1744 (type: Regular File) : [listing of many directory entries] : 34475 = 9388 (type: Regular File) 34955 = 10373 (type: Regular File) 28395 = 2370 (type: Regular File) Indirect blocks: 0 L1 0:30037a00:400 20000L/400P F=5 B=2246/2246 cksum=952b158ae3:48309161f6e2:14863f0c3647dc:4689b429072f201 0 L0 0:8002b400:200 4000L/200P F=1 B=2246/2246 cksum=12114b61b1:78931eb2743:198d94f2f4b5e:3ab0b9a86f3c18 4000 L0 0:8001e800:1600 4000L/1600P F=1 B=2244/2244 cksum=1321a384bb3:34b1dafdd5595:5eb3345070e37f4:7e9ba1615b5ce75 8000 L0 0:8002b600:1400 4000L/1400P F=1 B=2246/2246 cksum=12368be1ad5:31de62aa1d99b:537e7b8d91e37b8:832aa38a26f08b4d c000 L0 0:70c80e00:1600 4000L/1600P F=1 B=1555/1555 cksum=1247d01dc5a:3b38fbafe005a:6fb375255e34ced:926381771742ea8b 10000 L0 0:70aa5a00:1400 4000L/1400P F=1 B=1236/1236 cksum=10cbedd48b9:2fe5bb1d703f5:514bc78624461fa:4adb094208689915 segment [0000000000000000, 0000000000014000) size 80K Object lvl iblk dblk dsize dnsize lsize %full type 950 1 128K 4K 0 512 4K 0.00 ZFS plain file 228 bonus System attributes dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 0 path /O/0/d11/27403 uid 0 gid 0 atime Wed Dec 31 19:00:00 1969 mtime Wed Dec 31 19:00:00 1969 ctime Wed Dec 31 19:00:00 1969 crtime Mon Nov 21 02:20:34 2022 gen 856 mode 107666 size 0 parent 672 links 1 pflags 800000000000 rdev 0x0000000000000000 SA xattrs: 76 bytes, 1 entries trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\013k\000\000\000\000\000\000 Indirect blocks: Object lvl iblk dblk dsize dnsize lsize %full type 982 1 128K 4K 0 512 4K 0.00 ZFS plain file 228 bonus System attributes dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 0 path /O/0/d11/27435 uid 0 gid 0 atime Wed Dec 31 19:00:00 1969 mtime Wed Dec 31 19:00:00 1969 ctime Wed Dec 31 19:00:00 1969 crtime Mon Nov 21 02:20:34 2022 gen 856 mode 107666 size 0 parent 672 links 1 pflags 800000000000 rdev 0x0000000000000000 SA xattrs: 76 bytes, 1 entries trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000+k\000\000\000\000\000\000 Indirect blocks: Object lvl iblk dblk dsize dnsize lsize %full type 1014 1 128K 4K 0 512 4K 0.00 ZFS plain file 228 bonus System attributes dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 0 path /O/0/d11/27883 uid 0 gid 0 atime Wed Dec 31 19:00:00 1969 mtime Wed Dec 31 19:00:00 1969 ctime Wed Dec 31 19:00:00 1969 crtime Mon Nov 21 02:20:35 2022 gen 856 mode 107666 size 0 parent 672 links 1 pflags 800000000000 rdev 0x0000000000000000 SA xattrs: 76 bytes, 1 entries trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353l\000\000\000\000\000\000 Indirect blocks: : [listing of dnodes continues] : Object lvl iblk dblk dsize dnsize lsize %full type 38985 1 128K 4K 0 512 4K 0.00 ZFS plain file 312 bonus System attributes dnode flags: USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED dnode maxblkid: 0 path /O/0/d11/44267 uid 0 gid 0 atime Mon Nov 21 02:27:17 2022 mtime Mon Nov 21 02:27:17 2022 ctime Mon Nov 21 02:27:17 2022 crtime Mon Nov 21 02:25:27 2022 gen 1076 mode 107666 size 0 parent 672 links 1 pflags 800000000000 rdev 0x0000000000000000 SA xattrs: 160 bytes, 2 entries trusted.lma = \010\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\353\254\000\000\000\000\000\000 trusted.fid = B#\000\000\002\000\000\000\0267\000\000\000\000\000\000\000\000\020\000\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000 Indirect blocks: sanity test_312: @@@@@@ FAIL: zfs_objid unset </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a>  FYI ^</p> Related LU-14692 Development Rank 1|hzz1s7: Rank (Obsolete) 9223372036854775807 Severity