https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6096 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run of review-ldiskfs: <a href="https://testing.hpdd.intel.com/test_sets/277e606e-976d-11e4-bafa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/277e606e-976d-11e4-bafa-5254006e85c2</a>.</p> <p>I note that several other recent similar failures have been marked as <a href="https://jira.whamcloud.com/browse/LU-3534" title="async update cross-MDTs" class="issue-link" data-issue-key="LU-3534"><del>LU-3534</del></a>.<br/> As I'm unsure of the reasoning for that and this one is seen on el7 I've raised it as new.<br/> Somebody more expert may decide it's a dup after looking it over.</p> <p>The sub-test test_17m failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsck -fnvd /dev/lvm-Role_MDS/P1 e2fsck 1.42.12.wc1 (15-Sep-2014) shadow-26vm8: check_blocks:2814: increase inode 32775 badness 0 to 1 shadow-26vm8: check_blocks:2814: increase inode 32776 badness 0 to 1 Pass 1: Checking inodes, blocks, and sizes Inode 32775, i_size is 0, should be 4096. Fix? no Inode 32776, i_size is 0, should be 4096. Fix? no </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity 17m</p> LU-6096

sanity test_17m: e2fsck Inode 32775, i_size is 0, should be 4096

Bug Blocker Resolved Fixed Alex Zhuravlev Maloo HB Thu, 8 Jan 2015 21:10:35 +0000 Wed, 2 Nov 2016 16:34:30 +0000 Wed, 5 Aug 2015 12:50:23 +0000 Lustre 2.7.0 Lustre 2.8.0 0 13 <p>This is just a regular review-ldiskfs test run, not even review-dne, and it means that the MDT filesystem is corrupt at this point in the test, which isn't very far along.</p> <p>Bob, could you please run this test locally with RHEL 7, and find out what files the affected inodes belong to (32775 and 32776 in this case). That can be done with something like the following to find the pathnames for those inodes, as well as dump the attributes for them:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c /dev/lvm-Role_MDS/P1 debugfs&gt; ncheck 32775 32776 debugfs&gt; stat &lt;32775&gt; debugfs&gt; stat &lt;32776&gt; </pre> </div></div> <p>Then we need to check why that subtest is causing problems, but this will at least narrow it down to a single subtest.</p> <p>Bob, any chance to run this test on RHEL7 again?</p> <p>not yet, hoping to get to it today</p> <p>doesn't seem to need el7 server to fail. I see the following in a local test of only sanity, 17m with el7 client on el6 servers. the exact error looks different, but it still fails. I may have something wrong in my local environment.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 11:05:36 (1421175936) create 512 short and long symlink files under /mnt/lustre/d17m.sanitym erase them Waiting for local destroys to complete recreate the 512 symlink files with a shorter string stop and checking mds1: e2fsck -fnvd /dev/sdb Stopping /mnt/mds1 (opts:) on centos2 pdsh@centos7: centos2: ssh exited with exit code 1 centos2: e2fsck 1.42.12.wc1 (15-Sep-2014) centos2: [QUOTA WARNING] Usage inconsistent for ID 0:actual (3624960, 1237) != expected (3899392, 2307) centos2: [QUOTA WARNING] Usage inconsistent for ID 0:actual (3624960, 1237) != expected (3899392, 2306) pdsh@centos7: centos2: ssh exited with exit code 4 Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Update quota info for quota type 0? no Update quota info for quota type 1? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** 1246 inodes used (1.25%, out of 100000) 6 non-contiguous files (0.5%) 1 non-contiguous directory (0.1%) # of inodes with ind/dind/tind blocks: 5/0/0 17580 blocks used (35.16%, out of 50000) 0 bad blocks 1 large file 98 regular files 115 directories 0 character device files 0 block device files 0 fifos 0 links 1024 symbolic links (512 fast symbolic links) 0 sockets ------------ 1237 files Starting mds1: /dev/sdb /mnt/mds1 Started lustre-MDT0000 sanity test_17m: @@@@@@ FAIL: e2fsck should not report error upon short/long symlink MDT: rc=4 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4677:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4708:error() = /usr/lib64/lustre/tests/sanity.sh:611:test_17m() = /usr/lib64/lustre/tests/test-framework.sh:4955:run_one() = /usr/lib64/lustre/tests/test-framework.sh:4992:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4809:run_test() = /usr/lib64/lustre/tests/sanity.sh:614:main() Dumping lctl log to /tmp/test_logs/2015-01-13/110504/sanity.test_17m.*.1421176025.log FAIL 17m (94s) resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 resend_count is set to 4 4 == sanity test complete, duration 119 sec ============================================================ 11:07:17 (1421176037) sanity: FAIL: test_17m e2fsck should not report error upon short/long symlink MDT: rc=4 debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck sanity returned 0 Finished at Tue Jan 13 11:07:31 PST 2015 in 147s /usr/lib64/lustre/tests/auster: completed with rc 0 </pre> </div></div> <p>I see a similar error with an el6.6 client. whatever problem I'm seeing isn't el7 only.</p> <p>all the rest of test 17 subtests work fine. it looks like only 17m has problems</p> <p>errors reported above I think were due to missing quota-devel rpm on all my local test nodes. with that rpm installed on client &amp; servers I'm no longer seeing any error in sanity, 17m. That's with el6 servers. Still trying to reproduce the original error in the ticket.</p> <p>I tried the simplest possible config, client &amp; server on one node. The problem reproduces.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 14:21:42 (1421187702) create 512 short and long symlink files under /mnt/lustre/d17m.sanitym erase them Waiting for local destroys to complete recreate the 512 symlink files with a shorter string stop and checking mds1: e2fsck -fnvd /dev/sdb Stopping /mnt/mds1 (opts:) on centos7 e2fsck 1.42.12.wc1 (15-Sep-2014) check_blocks:2814: increase inode 32777 badness 0 to 1 check_blocks:2814: increase inode 32779 badness 0 to 1 Pass 1: Checking inodes, blocks, and sizes Inode 32777, i_size is 0, should be 4096. Fix? no Inode 32779, i_size is 0, should be 4096. Fix? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** </pre> </div></div> <p>debugfs output</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos7 bogl]# umount /dev/sdb [root@centos7 bogl]# debugfs /dev/sdb debugfs 1.42.12.wc1 (15-Sep-2014) debugfs: ncheck 32777 32779 Inode Pathname 32779 //PENDING 32777 /ROOT/.lustre/fid debugfs: stat PENDING Inode: 32779 Type: directory Mode: 0755 Flags: 0x0 Generation: 512259630 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 0 File ACL: 0 Directory ACL: 0 Links: 2 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 atime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 mtime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 crtime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "00 00 00 00 00 00 00 00 03 00 00 00 02 00 00 00 0a 00 00 00 00 00 00 00 " (24) lma: fid=[0x200000003:0xa:0x0] compat=0 incompat=0 BLOCKS: (0):16620 TOTAL: 1 debugfs: stat /ROOT/.lustre/fid Inode: 32777 Type: directory Mode: 0100 Flags: 0x0 Generation: 512259628 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 0 File ACL: 0 Directory ACL: 0 Links: 2 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 atime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 mtime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 crtime: 0x54b599b4:7dbd4a3c -- Tue Jan 13 14:18:28 2015 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "00 00 00 00 00 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00 00 00 00 00 " (24) lma: fid=[0x200000002:0x2:0x0] compat=0 incompat=0 BLOCKS: (0):16618 TOTAL: 1 </pre> </div></div> <p>non-zero Blockcount with 0 size on those entries looks kind of suspicious, but what do I know <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The client version doesn't matter, only the server version, since clients don't interact with the soak filesystem directly. </p> <p>The affected files (PENDING, .lustre/fid) are internally created, so this is a bug in the osd-ldiskfs or MDD code. </p> <p>Bob, do I understand correctly that you can reproduce the issue with el7 server only? I can't reproduce with el6 locally.</p> <p>Alex, yes exactly. Appears 100% reproducible but only with el7 server.</p> <p>then can I guess this is introduced with one of the patches adding el7 support?</p> <p>that's possible but not certain. el7 ldiskfs is <a href="http://review.whamcloud.com/#/c/10249" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10249</a></p> <p>could be some other non-obvious diff in kernel internal API for vfs, for example<br/> even some e2fsck flaw specific to el7 is possible</p> <p>oh, yes, for sure.. we know, linux kernel api is super stable <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Actually I also have seen this error as well.</p> <p>James, with el7 only?</p> <p>Got around to update the SLES12 ldiskfs support to see if I can reproduce this issue. Before I saw this issue but now I can't seem to reproduce it with the latest ldiskfs patch.</p> <p>Is it still happening on EL7?</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14469" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14469</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6096" title="sanity test_17m: e2fsck Inode 32775, i_size is 0, should be 4096" class="issue-link" data-issue-key="LU-6096"><del>LU-6096</del></a> test: force test of known failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 747eeed7e4eec9ea5c900886d97709b7d9ef9e0b</p> <p>quickest way to prove it's still a problem is to let autotest tell us</p> <p>yes, it still fails. see <a href="https://testing.hpdd.intel.com/test_sets/32d9d04a-e38c-11e4-bb64-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32d9d04a-e38c-11e4-bb64-5254006e85c2</a></p> <p>Alex, can you please take a look at this?</p> <p>I pushed a debug patch.</p> <p>I think there may be a typo in that debug patch. title in commit header refers to 6069, not 6096</p> <p>ok, thanks. that's just to try to catch the cause.. not to fix the issue.</p> <p>Patch is <a href="http://review.whamcloud.com/14772" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14772</a></p> <p>It looks like the requested sanity test_17m test was run, but none of the added LASSERTs/BUG were hit.</p> <p>Alex, what is the next step?</p> <p>going to reproduce locally, will take a bit.</p> <p>the patch referenced above should address the issue.</p> <p>In local test on el7 servers the fix works. The problem which was 100% reproducible now doesn't appear even once.</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/14772/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14772/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6096" title="sanity test_17m: e2fsck Inode 32775, i_size is 0, should be 4096" class="issue-link" data-issue-key="LU-6096"><del>LU-6096</del></a> osd: mark directory's inode dirty<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 52437f5218fc911cc0b6b9f36e8a143bf54f1024</p> <p>Patch landed to master for 2.8.0.</p> <p>Hit this error again on build #3071 EL7 DNE mode</p> <p><a href="https://testing.hpdd.intel.com/test_sets/683aadc2-1311-11e5-8d21-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/683aadc2-1311-11e5-8d21-5254006e85c2</a></p> <p>It looks like this is still being hit. The symptom is slightly different - instead of only the <tt>PENDING</tt> and <tt>.lustre/fid</tt> directories being affected, it looks like a larger number of directories are having problems:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>onyx-41vm7: e2fsck 1.42.12.wc1 (15-Sep-2014) onyx-41vm7: check_blocks:2814: increase inode 139 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 140 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 142 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 143 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 145 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 146 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 148 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524394 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524397 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524402 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524405 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524408 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524411 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524416 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524421 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524424 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524429 badness 0 to 1 onyx-41vm7: check_blocks:2814: increase inode 524433 badness 0 to 1 Pass 1: Checking inodes, blocks, and sizes Inode 139, i_size is 0, should be 4096. Fix? no Inode 140, i_size is 0, should be 4096. Fix? no Inode 142, i_size is 0, should be 4096. Fix? no Inode 143, i_size is 0, should be 4096. Fix? no Inode 145, i_size is 0, should be 4096. Fix? no Inode 146, i_size is 0, should be 4096. Fix? no Inode 148, i_size is 0, should be 4096. Fix? no Inode 524394, i_size is 0, should be 4096. Fix? no Inode 524397, i_size is 0, should be 4096. Fix? no Inode 524402, i_size is 0, should be 4096. Fix? no Inode 524405, i_size is 0, should be 4096. Fix? no Inode 524408, i_size is 0, should be 4096. Fix? no Inode 524411, i_size is 0, should be 4096. Fix? no Inode 524416, i_size is 0, should be 4096. Fix? no Inode 524421, i_size is 0, should be 4096. Fix? no Inode 524424, i_size is 0, should be 4096. Fix? no Inode 524429, i_size is 0, should be 4096. Fix? no Inode 524433, i_size is 0, should be 4096. Fix? no </pre> </div></div> <p>It still always seems related to EL7 though.</p> <p>Alex, any comment on this? Is it possible the filesystem is being uncounted without flushing those updates to disk?</p> <p>I'm trying to reproduce this locally.. will take some time.</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15581" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15581</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6096" title="sanity test_17m: e2fsck Inode 32775, i_size is 0, should be 4096" class="issue-link" data-issue-key="LU-6096"><del>LU-6096</del></a> osd: mark dir's inode dirty<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d62ecb63fb898caa8c83f99ea325d5ad3fb96e00</p> <p>Bob is this also a problem for SLES12?</p> <p>James, have never seen it on sles12, only on el7. can't promise it's not in sles12 too.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15581/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15581/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6096" title="sanity test_17m: e2fsck Inode 32775, i_size is 0, should be 4096" class="issue-link" data-issue-key="LU-6096"><del>LU-6096</del></a> ldiskfs: mark dir's inode dirty<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bad49e39e301d4367eaead5ee566f5dcacfde8f6</p> <p>Landed for 2.8. Resolving ticket. If this every crops up on SLES12 we can always open a ticket for it then</p> Duplicate LU-7094 Related LU-5022 Development Rank 1|hzx3hr: Rank (Obsolete) 16971 Severity