https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1015 Lustre <p>We have been running ior testing on hyperion with toss 5 and have seen ldiskfs corruption. Since I know you have access to hyperion, I was hoping you could log on and look around (the console logs are on hyperion577-pub and santricity can be run from there as well). I have set up a test filesystem called /p/ls1 created with large luns (22TB per lun with 6 luns on each RBOD) on a Netapp. The mds is on hyperion-agb25 and the 2 oss nodes are hyperion-agb27 and hyperion-agb28. I had 10 clients writing i/o to the filesystem and would power cycle an oss every hour to simulate a node crashing. Upon bringing the oss up I would run the full fsck to check for errors and bring up lustre again and continue the i/o load from clients. We hit a bug where the fsck shows corruption and doesn't mount lustre. As a side note, I was running the same testing in parallel with the same HW, but with a small 3TB lun size and did not hit this issue. </p> <p>zgrep Mounting ../conman.old/console.hyperion-agb27-20120115.gz</p> <p>2012-01-14 14:24:02 Mounting /dev/dm-3 on /mnt/lustre/local/ls1-OST0000<br/> 2012-01-14 14:24:04 Mounting /dev/dm-0 on /mnt/lustre/local/ls1-OST0001<br/> 2012-01-14 14:24:06 Mounting /dev/dm-4 on /mnt/lustre/local/ls1-OST0002<br/> 2012-01-14 15:21:57 Mounting local filesystems: [ OK ]<br/> 2012-01-14 15:22:03 Mounting other filesystems: [ OK ]<br/> 2012-01-14 15:24:13 Mounting /dev/dm-0 on /mnt/lustre/local/ls1-OST0000<br/> 2012-01-14 15:24:15 Mounting /dev/dm-3 on /mnt/lustre/local/ls1-OST0001<br/> 2012-01-14 15:24:17 Mounting /dev/dm-1 on /mnt/lustre/local/ls1-OST0002<br/> 2012-01-14 16:21:49 Mounting local filesystems: [ OK ]<br/> 2012-01-14 16:21:55 Mounting other filesystems: [ OK ]<br/> 2012-01-14 16:23:51 Mounting /dev/dm-3 on /mnt/lustre/local/ls1-OST0000<br/> 2012-01-14 16:23:53 Mounting /dev/dm-1 on /mnt/lustre/local/ls1-OST0001<br/> 2012-01-14 16:23:55 Mounting /dev/dm-5 on /mnt/lustre/local/ls1-OST0002<br/> 2012-01-14 17:21:53 Mounting local filesystems: [ OK ]<br/> 2012-01-14 17:21:59 Mounting other filesystems: [ OK ]<br/> 2012-01-14 18:22:00 Mounting local filesystems: [ OK ]<br/> 2012-01-14 18:22:06 Mounting other filesystems: [ OK ]<br/> 2012-01-14 19:21:56 Mounting local filesystems: [ OK ]<br/> 2012-01-14 19:22:02 Mounting other filesystems: [ OK ]<br/> 2012-01-14 20:21:52 Mounting local filesystems: [ OK ]<br/> 2012-01-14 20:21:58 Mounting other filesystems: [ OK ]</p> <p>It appears that the corruption occurred way back on Friday 1/14 after 16:20.<br/> I state this based upon the fact that the OST's did not make it back after the power cycle on 1/14 @ 17:20.</p> <p>Also the following fsck results only surfaced after that power cycle:</p> <p>2012-01-14 17:23:48 Group descriptor 0 checksum is invalid. FIXED.<br/> 2012-01-14 17:23:48 Group descriptor 1 checksum is invalid. FIXED.<br/> 2012-01-14 17:23:48 Group descriptor 2 checksum is invalid. FIXED.</p> lustre-2.1.0-13chaos_2.6.32_220.1chaos.ch5.x86_64.x86_64<br/> toss/chaos 5<br/> NetApp 22TB LUNs LU-1015

ldiskfs corruption with large LUNs

Bug Blocker Resolved Fixed Andreas Dilger cindy heer ldiskfs paj Wed, 18 Jan 2012 12:24:52 +0000 Mon, 11 Jun 2012 22:40:04 +0000 Mon, 11 Jun 2012 22:40:04 +0000 Lustre 2.3.0 Lustre 2.1.1 Lustre 2.1.2 0 12 <p>Andreas</p> <p>Can you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>I don't have Hyperion access myself, unfortunately.</p> <p>Are there any ldiskfs errors on the OSS consoles between 16:23, when the OSTs last successfully mounted, and 17:21, when they apparently failed to mount?</p> <p>Did all of the OSTs fail in a similar manner, or just a single one?</p> <p>Are the group descriptor checksum error messages just the first of many (i.e. are all the group checksums invalid), or is it only for groups 0, 1, 2?</p> <p>Could the OST(s) be mounted after the e2fsck run fixed those checksum errors, or were there other errors/corruption that prevented the OST(s) from mounting?</p> <p>It looks like you are running the RHEL6.2 (220) kernel, so this should be relatively up-to-date w.r.t. upstream ext4 patches, or at least would hopefully narrow down the number of upstream kernel patches to look at.</p> <p>Syslog from 13:20 to ~17:30 2012-01-14</p> <p>I have access, so i took the liberty. <br/> I do see these errors throughout:<br/> 2012-01-14 04:22:21 Buffer I/O error on device sdj, logical block 1</p> <p>On 2012-01-14 the first correctable pfsck errors appear at the 13:23 pfsck.<br/> At the 14:23 pfsck correctable errors appear on two disks.<br/> The 15:23 and 16:23 pfsck are clean.<br/> At 17:23, things explode</p> <p>2012-01-14 17:23:47 ls1-OST0002: recovering journal<br/> 2012-01-14 17:23:48 fsck.ldiskfs: Group descriptors look bad... trying backup blocks...<br/> 2012-01-14 17:23:48 One or more block group descriptor checksums are invalid. Fix? yes<br/> 2012-01-14 17:23:48<br/> Followed by a large mess of errors. <br/> I don't see any odd error messages to account for this, is it possible the device was dual-mounted?<br/> Is the hardware healthy? The syslog from 13:23 to the failure is attached.</p> <p>Thanks for looking around. I'm currently trying to build another test <br/> with ddn hardware as a comparison. The device was not dual mounted and <br/> the Netapp hardware exhibits no errors. </p> <p>This corruption occurs as a result of unclean shutdowns of the OSS.</p> <p>No, there are no errors from ldiskfs before this occurs. The disk state is inconsistent after the unclean OSS shutdown. If allowed to run without a full fsck, it will result in ldiskfs panicing the node.</p> <p>An preen fsck (fsck -p, which is the default in our init scripts) will not necessarily catch this. If it fails to catch it and the node starts up, ldiskfs will panic the node later.</p> <p>No, not all OSTs hit this. It is more-or-less random which OSTs will be corrupt after the unclean shutdown.</p> <p>Yes, so far fsck will fix the problems and allow ldiskfs to run without error. But we haven't looked too hard to make sure there was no data loss. I think some times we I did wind up with files in lost+found.</p> <p>Cindy, if you aren't already, after powering off the nodes you'll want to avoid using the default "fsck.ldiskfs -p" and instead do "fsck.ldiskfs -f -n" to really find the errors, and to avoid fixing them before Whamcloud can look at them.</p> <p>Actually, there WAS one time that I hosed an OST so badly that I gave up and just reformatted it (I was trying to make progress on <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> at the time). But I'm not sure of the steps involved there...it may have been something like:</p> <p> 1) unclean power-off<br/> 2) fsck -p (fix something, but not everything?)<br/> 3) start ldiskfs, run for a while<br/> 4) ldiskfs panics<br/> 5) reboot and manually run fsck -f -y, something not quite right...<br/> 6) give up and format</p> <p>I have restarted testing on ls1 filesystem on hyperion (large lun testing with netapp) with the fsck -n in place. I was not running fsck -p for this testing ever, but I did have the full fsck with -y in the past. That has been modified. I am also running the same testing on ls3 filesystem on hyperion (large luns with ddn hardware) that has been running for about a week and has not had any failures so far. I continue to run iors to the filesystems while every hour I simulate an unclean shutdown on the oss (by powering them off). </p> <p>I think I hit a bit of corruption again on the large LUN with Netapp (still no corruption exhibited with the DDN). Here is the output (after oss is powered off to simulate oss crash). I'm running a journal replay fsck.ldiskfs -p and then a full fsck with -n flag): </p> <p>ldev fsck.ldiskfs -p %d<br/> ls1-OST0001: ls1-OST0001: clean, 337/22888320 files, 26023621/5859409133 blocks<br/> ls1-OST0000: ls1-OST0000 contains a file system with errors, check forced.<br/> ls1-OST0002: ls1-OST0002: clean, 340/22888320 files, 24807109/5859409133 blocks<br/> ls1-OST0000: ls1-OST0000: Duplicate or bad block in use!<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 1769473: 4747954688<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 1769474: 4747984896 4747984897 4747984898<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 1769475: 4747988992 4747990016 4747990017<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 1769476: 4747986944 4747987968 4747987969<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 18546689: 4747954688<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 18546690: 4747984896 4747984897 4747984898<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 18546691: 4747988992 4747990016 4747990017<br/> ls1-OST0000: ls1-OST0000: Multiply-claimed block(s) in inode 18546692: 4747986944 4747987968 4747987969<br/> ls1-OST0000: ls1-OST0000: (There are 8 inodes containing multiply-claimed blocks.)<br/> ls1-OST0000: <br/> ls1-OST0000: ls1-OST0000: File /lost+found/#1769473 (inode #1769473, mod time Tue Jan 31 09:39:00 2012) <br/> ls1-OST0000: has 1 multiply-claimed block(s), shared with 1 file(s):<br/> ls1-OST0000: ls1-OST0000: /CONFIGS (inode #18546689, mod time Tue Jan 31 09:39:00 2012)<br/> ls1-OST0000: ls1-OST0000: <br/> ls1-OST0000: <br/> ls1-OST0000: ls1-OST0000: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.<br/> ls1-OST0000: (i.e., without -a or -p options)<br/> ldev: Fatal: parallel command execution failed</p> <p>e-agb27: pfsck.ldiskfs /dev/dm-1 /dev/dm-2 /dev/dm-0 &#8211; -f -n -v -t</p> <p>e-agb27: fsck 1.41.90.1chaos (14-May-2011)<br/> e-agb27: fsck.ldiskfs 1.41.90.1chaos (14-May-2011)<br/> e-agb27: fsck.ldiskfs 1.41.90.1chaos (14-May-2011)</p> <p>e-agb27: fsck.ldiskfs 1.41.90.1chaos (14-May-2011)<br/> e-agb27: Pass 1: Checking inodes, blocks, and sizes</p> <p>e-agb27: Pass 1: Checking inodes, blocks, and sizes<br/> e-agb27: Pass 1: Checking inodes, blocks, and sizes</p> <p>e-agb27: Pass 2: Checking directory structure</p> <p>e-agb27: Pass 3: Checking directory connectivity<br/> e-agb27: Pass 4: Checking reference counts</p> <p>e-agb27: Pass 5: Checking group summary information</p> <p>e-agb27: Pass 2: Checking directory structure</p> <p>e-agb27: Pass 3: Checking directory connectivity<br/> e-agb27: Pass 4: Checking reference counts</p> <p>e-agb27: <br/> e-agb27: Running additional passes to resolve blocks claimed by more than one inode...<br/> e-agb27: Pass 1B: Rescanning for multiply-claimed blocks<br/> e-agb27: Multiply-claimed block(s) in inode 1769473: 4747954688<br/> e-agb27: Multiply-claimed block(s) in inode 1769474: 4747984896 4747984897 4747984898<br/> e-agb27: Multiply-claimed block(s) in inode 1769475: 4747988992 4747990016 4747990017<br/> e-agb27: Multiply-claimed block(s) in inode 1769476: 4747986944 4747987968 4747987969<br/> e-agb27: Pass 5: Checking group summary information<br/> e-agb27: Multiply-claimed block(s) in inode 18546689: 4747954688<br/> e-agb27: Multiply-claimed block(s) in inode 18546690: 4747984896 4747984897 4747984898<br/> e-agb27: Multiply-claimed block(s) in inode 18546691: 4747988992 4747990016 4747990017<br/> e-agb27: Multiply-claimed block(s) in inode 18546692: 4747986944 4747987968 4747987969<br/> e-agb27: Pass 1C: Scanning directories for inodes with multiply-claimed blocks</p> <p>e-agb27: Pass 1D: Reconciling multiply-claimed blocks<br/> e-agb27: <br/> e-agb27: 337 inodes used (0.00%)<br/> e-agb27: 131 non-contiguous files (38.9%)<br/> e-agb27: 0 non-contiguous directories (0.0%)<br/> e-agb27: # of inodes with ind/dind/tind blocks: 0/0/0<br/> e-agb27: Extent depth histogram: 178/149<br/> e-agb27: 26023621 blocks used (0.44%)<br/> e-agb27: 0 bad blocks<br/> e-agb27: 19 large files<br/> e-agb27: <br/> e-agb27: 291 regular files<br/> e-agb27: 37 directories<br/> e-agb27: 0 character device files<br/> e-agb27: 0 block device files<br/> e-agb27: 0 fifos<br/> e-agb27: 0 links<br/> e-agb27: 0 symbolic links (0 fast symbolic links)<br/> e-agb27: 0 sockets<br/> e-agb27: --------<br/> e-agb27: 328 files<br/> e-agb27: Memory used: 43816k/715264k (16113k/27704k), time: 7.44/ 6.65/ 0.75<br/> e-agb27: I/O read: 29MB, write: 0MB, rate: 3.90MB/s</p> <p>e-agb27: <br/> e-agb27: 340 inodes used (0.00%)<br/> e-agb27: 132 non-contiguous files (38.8%)<br/> e-agb27: 0 non-contiguous directories (0.0%)<br/> e-agb27: # of inodes with ind/dind/tind blocks: 0/0/0<br/> e-agb27: Extent depth histogram: 180/150<br/> e-agb27: 24807109 blocks used (0.42%)<br/> e-agb27: 0 bad blocks<br/> e-agb27: 18 large files<br/> e-agb27: <br/> e-agb27: 294 regular files<br/> e-agb27: 37 directories<br/> e-agb27: 0 character device files<br/> e-agb27: 0 block device files<br/> e-agb27: 0 fifos<br/> e-agb27: 0 links<br/> e-agb27: 0 symbolic links (0 fast symbolic links)<br/> e-agb27: 0 sockets<br/> e-agb27: --------<br/> e-agb27: 331 files<br/> e-agb27: Memory used: 43816k/715264k (16113k/27704k), time: 15.80/ 6.73/ 0.76<br/> e-agb27: I/O read: 30MB, write: 0MB, rate: 1.90MB/s</p> <p>e-agb27: (There are 8 inodes containing multiply-claimed blocks.)<br/> e-agb27: <br/> e-agb27: File /lost+found/#1769473 (inode #1769473, mod time Tue Jan 31 09:39:00 2012) <br/> e-agb27: has 1 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /CONFIGS (inode #18546689, mod time Tue Jan 31 09:39:00 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File /lost+found/#1769474 (inode #1769474, mod time Wed Jan 11 12:54:06 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /???/mountdata (inode #18546690, mod time Wed Jan 11 12:54:06 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File ... (inode #1769475, mod time Tue Jan 31 09:39:00 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /???/ls1-OST0000 (inode #18546691, mod time Tue Jan 31 09:39:00 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File /lost+found/#1769476 (inode #1769476, mod time Sat Jan 14 16:23:53 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: ... (inode #18546692, mod time Sat Jan 14 16:23:53 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File /CONFIGS (inode #18546689, mod time Tue Jan 31 09:39:00 2012) <br/> e-agb27: has 1 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /lost+found/#1769473 (inode #1769473, mod time Tue Jan 31 09:39:00 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File /???/mountdata (inode #18546690, mod time Wed Jan 11 12:54:06 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /lost+found/#1769474 (inode #1769474, mod time Wed Jan 11 12:54:06 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File /???/ls1-OST0000 (inode #18546691, mod time Tue Jan 31 09:39:00 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: ... (inode #1769475, mod time Tue Jan 31 09:39:00 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: File ... (inode #18546692, mod time Sat Jan 14 16:23:53 2012) <br/> e-agb27: has 3 multiply-claimed block(s), shared with 1 file(s):<br/> e-agb27: /lost+found/#1769476 (inode #1769476, mod time Sat Jan 14 16:23:53 2012)<br/> e-agb27: Clone multiply-claimed blocks? no<br/> e-agb27: <br/> e-agb27: Delete file? no<br/> e-agb27: <br/> e-agb27: Pass 2: Checking directory structure<br/> e-agb27: Invalid inode number for '.' in directory inode 1769473.<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Pass 3: Checking directory connectivity<br/> e-agb27: '..' in /lost+found/#1769473 (1769473) is / (2), should be /lost+found (11).<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Pass 4: Checking reference counts<br/> e-agb27: Inode 2 ref count is 5, should be 6. Fix? no<br/> e-agb27: <br/> e-agb27: Inode 11 ref count is 3, should be 2. Fix? no<br/> e-agb27: <br/> e-agb27: Inode 1769473 ref count is 2, should be 1. Fix? no<br/> e-agb27: <br/> e-agb27: Inode 1769474 ref count is 2, should be 1. Fix? no<br/> e-agb27: <br/> e-agb27: Unattached inode 1769475<br/> e-agb27: Connect to /lost+found? no<br/> e-agb27: <br/> e-agb27: Inode 18546689 ref count is 2, should be 3. Fix? no<br/> e-agb27: <br/> e-agb27: Inode 18546691 ref count is 1, should be 2. Fix? no<br/> e-agb27: <br/> e-agb27: Unattached inode 18546692<br/> e-agb27: Connect to /lost+found? no<br/> e-agb27: <br/> e-agb27: Unattached zero-length inode 18546693. Clear? no<br/> e-agb27: <br/> e-agb27: Unattached inode 18546693<br/> e-agb27: Connect to /lost+found? no<br/> e-agb27: <br/> e-agb27: Inode 18546695 ref count is 1, should be 2. Fix? no<br/> e-agb27: <br/> e-agb27: Pass 5: Checking group summary information</p> <p>e-agb27: Block bitmap differences: <del>(6385</del><del>6391) -(453017600</del><del>453017602) -(453018113</del><del>453018114) -(453018626</del><del>453018627) -453019136 -453019648 -(453020162</del><del>453020416) -(453020672</del><del>453020673) -453021187 -453021696 -(453022720</del>-453022721)<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Free blocks count wrong for group #13825 (32768, counted=32498).<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Free blocks count wrong (5835336736, counted=5835336466).<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Inode bitmap differences: -1769477 -1769479<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Free inodes count wrong for group #13824 (125, counted=122).<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: Free inodes count wrong (22887977, counted=22887974).<br/> e-agb27: Fix? no<br/> e-agb27: <br/> e-agb27: <br/> e-agb27: ls1-OST0000: ********** WARNING: Filesystem still has errors **********<br/> e-agb27: <br/> e-agb27: <br/> e-agb27: 343 inodes used (0.00%)<br/> e-agb27: 135 non-contiguous files (39.4%)<br/> e-agb27: 0 non-contiguous directories (0.0%)<br/> e-agb27: # of inodes with ind/dind/tind blocks: 0/0/0<br/> e-agb27: Extent depth histogram: 184/150<br/> e-agb27: 24072397 blocks used (0.41%)<br/> e-agb27: 0 bad blocks<br/> e-agb27: 18 large files<br/> e-agb27: <br/> e-agb27: 297 regular files<br/> e-agb27: 38 directories<br/> e-agb27: 0 character device files<br/> e-agb27: 0 block device files<br/> e-agb27: 0 fifos<br/> e-agb27: 3 links<br/> e-agb27: 0 symbolic links (0 fast symbolic links)<br/> e-agb27: 0 sockets<br/> e-agb27: --------<br/> e-agb27: 335 files<br/> e-agb27: Memory used: 46616k/1430528k (16114k/30503k), time: 142.55/129.36/ 1.75<br/> e-agb27: I/O read: 721MB, write: 0MB, rate: 5.06MB/s<br/> e-agb27: FAILED: pfsck.ldiskfs &#8211; -f -n -v -t: 0</p> <p>I will leave the oss (hyerion-agb27) down for further examination unless I hear otherwise. </p> <p>Cindy, thanks for posting the e2fsck output. Looking at the blocks that are duplicate allocated:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e-agb27: Multiply-claimed block(s) in inode 1769473: 4747954688 e-agb27: Multiply-claimed block(s) in inode 1769474: 4747984896 4747984897 4747984898 e-agb27: Multiply-claimed block(s) in inode 1769475: 4747988992 4747990016 4747990017 e-agb27: Multiply-claimed block(s) in inode 1769476: 4747986944 4747987968 4747987969 </pre> </div></div> <p>These are all beyond the 2^32 block limit (4747986944 = 0x11b008800) so it may be that this problem relates to overflow of 32-bit block numbers somewhere in the IO stack. If you have the logs from the previous e2fsck runs that showed corruption, it would be useful to know if the type of corruption always the same or not (attaching a few e2fsck logs would be useful). I have a suspicion based on this one log that it may relate to corruption of the block bitmap during the <em>previous</em> journal recovery or e2fsck (possibly due to 2^32-block truncation) which later causes the blocks to be allocated twice.</p> <p>Several things can be done to begin debugging this, possibly in parallel:</p> <ul> <li>Verify IO integrity for &gt; 16TB LUNs</li> </ul> <ul class="alternate" type="square"> <li>take a &gt; 16TB OST out of service</li> <li>mount it directly with ldiskfs</li> <li>run "llverfs -v -l -c 32 {mountpoint} <p>" against the mounted OST filesystem</p></li> <li>run a full e2fsck to determine if there is any corruption present</li> </ul> <ul> <li>run the current test, but dump the journal contents before mounting or running e2fsck on the filesystem</li> </ul> <ul class="alternate" type="square"> <li>dumpe2fs $ostdev &gt; $logdir/$ostdev.$(date +%Y%m%d).stats</li> <li>debugfs -c -R "dump &lt;8&gt;" $ostdev &gt; $logdir/$ostdev.$(date +%y%m%d).journal</li> <li>debugfs -c -R "logdump -a" $ostdev &gt; $logdir/$ostdev.$(date +%Y%m%d).logdump</li> <li>e2fsck -fp $ostdev 2&gt;&amp;1 | tee $logdir/$ostdev.$(date +%Y%m%d).e2fsck</li> <li>dumpe2fs $ostdev &gt; $logdir/$ostdev.$(date +%Y%m%d).stats.post</li> </ul> <ul> <li>format some OSTs to be less than 16TB and try to reproduce the problem again on the NetApp hardware</li> </ul> <p>llverfs is a non-destructive test that will write files until the device is full and then read them back and verify the data contents. It is intended to catch errors in the IO path (filesystem, block layer, HBA, driver, controller) related to 32-bit address truncation, but it is not very fast (may take a couple of days, depending on LUN size and IO rate). If this finds problems, there is a lower-level (filesystem destructive) "llverdev" tool that will run against the underlying block device and do a full write/read/verify cycle on the device, excluding the filesystem. I don't have high hopes for this finding a problem, but it is useful to eliminate the chance that there are obvious bugs in the IO stack. We ran full llverdev and llverfs tests previously with a DDN SFA10kE + RHEL5 on a 128TB LUN without problems, but there may be problems with RHEL6, the driver, the controller, etc. in your environment.</p> <p>The debugfs/e2fsck commands are intended to catch (or at least give us some chance to find post-facto) errors in the journal replay and/or e2fsck that are incorrectly marking blocks free, which are later being reallocated. Also, what version of e2fsprogs is e2fsck-1.41.90-1chaos based on? I now recall after working on this bug that there may have been some 64-bit bugs fixed in e2fsck that could potentially be causing problems as well.</p> <p>Andreas, thanks for the excellent analysis. We have been running this same test to isolate the extent of the corruption. We see this behavior on the 22TB luns on the NetApp hardware, but not on a smaller 3TB partition created on similar luns. We also have not seen it on a DDN SFA10K with 16TB luns, but are reconfiguring the DDN to have luns &gt; 16TB to see if we can reproduce it on that hardware.</p> <p>Whamcloud has access to Hyperion, so please coordinate with the Hyperion team to reserve some hardware, and your folks can run the tests you describe above.</p> <p>Setup with 4 OSTS</p> <p>/dev/sda 12T 39G 12T 1% /p/osta<br/> /dev/sdb 22T 39G 21T 1% /p/ostb<br/> /dev/sdc 10T 39G 9.5T 1% /p/ostc<br/> /dev/sdd 22T 39G 21T 1% /p/ostd</p> <p>Ran llverfs per Andreas on /dev/sdd - no issues. <br/> Mounted all as Lustre FS, ran IOR from 3 clients, powercycled e-abg26 in the middle of the run,<br/> replicated failure. Captured before/after data in script per Andreas instructions, attached.</p> <p>Multiple files for size reason</p> <p>Before run</p> <p>small files</p> <p>before run</p> <p>after fail</p> <p>after fail</p> <p>Ran test on all disks, only sdd showed failure (so far)</p> <p>Repeated test, sdb (2nd 21TB LUN) failed this time. Seems easy to repeat, awaiting further requests.</p> <p>Cliff, some questions:</p> <ul class="alternate" type="square"> <li>presumably the 4 LUNs (12 TB, 22TB, 10TB, 22TB) are on the same node?</li> <li>you are not doing failover between multiple OSS nodes, but just rebooting the node in-place?</li> <li>how many times did you run the test, and how many failures? Two tests and two failures?</li> <li>this is running on the NetApp hardware for both the under 16TB and over 16TB LUNs?</li> </ul> <p>Hitting three failures on the &gt; 16TB LUNs is a fairly good indication of the problem is limited to &gt; 16TB LUN support, and not to the NetApp itself (assuming the &lt; 16TB LUNs are also on the NetApp). One option would be to run IOR directly on the OSS node against one of the &gt; 16TB LUNs mounted with "-t ldiskfs" instead of "-t lustre", then do a similar hard reset + e2fsck (and other debugging, which is hopefully in a script by now). This would let us see if this is a problem in the lustre/obdfilter/fsfilt code or in the core ldiskfs code. If this local testing still fails with ldiskfs, then it would be useful to test with ext4 to determine if the problem is in the base ext4 code.</p> <p>Look at the debug logs for the first test, and it does appear that the corruption is of the block bitmap above the 16TB mark, allowing the writes to reallocate those blocks. It may be in ldiskfs or ext4, so starting the above testing would also help cut down the number of variables.</p> <p>Per LLNL: all disk are from Netapp “E5400 RBODs” All 4 LUNS are attached to a single OSS node. <br/> I did not do failover, as there was nothing to fail to, I performed a hard powercycle using powerman. <br/> The test was run twice, the debug data script was run prior to the first test. Two tests, two failures. </p> <p>Again, afaik these LUNS are all furnished by the same NetApp device.</p> <p>I will perform the local IOR test and report results. </p> <p>Ran the test using IOR on a local ldiskfs mount. Failed. Results attached.<br/> Reformatted all LUNS as ext4, re-ran test twice - no failures. The second time, ran IOR in a loop against all four drives (in sequence) for one hour prior to powercycle. </p> <p>Failure with local ldiskfs mount.</p> <p>Options for Lustre format:</p> <p>mkfs.lustre --reformat --ost --fsname lu1015 --mgsnode=192.168.120.25@o2ib --mkfsoptions='-t ext4 -J size=2048 -O extents -G 256 -i 69905' /dev/sd$i &amp;</p> <p>Cliff, is this running the LLNL ldiskfs RPM, or the ldiskfs from the Lustre tree? It would be good to run vanilla ext4 powerfail tests a couple of extra times to more positively verify that the bug is not present with ext4, since we know that it is definitely still there for ldiskfs.</p> <p>rpm -qa |grep disk<br/> lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.gddd1a7c.x86_64_g0203c14.x86_64<br/> lustre-ldiskfs-debuginfo-3.3.0-2.6.32_220.4.2.el6_lustre.gddd1a7c.x86_64_g0203c14.x86_64</p> <p>I will continue running the powerfail test on ext4.</p> <p>Somewhat of a head-scratcher atm. Wanted to be certain only the large disks were seeing errors when formatted as ldiskfs, so last night ran a test writing to all four disks mounted as ldiskfs in a loop (short iteration of IOR) creating a new file each loop. <br/> I let the test run for one hour before doing the power cycle, previously had powercycled within 10 minutes of test start.<br/> First reboot - no disks! - restarted ibsrp via /etc/init.d/ibsrp - no errors found.<br/> Second reboot as a test, I restarted ibsrp by hand prior to running the fsck script, again no errors found. <br/> Not sure what to make of this outcome. </p> <p>Today, will re-run ext4 tests, so far no errors there. </p> <p>I have continued running the local (ext4 and ldiskfs) tests, but have not had a failure in two days.<br/> Am uncertain what if anything has changed. </p> <p>I thought of another possible way to positively exclude the NetApp from the picture here:</p> <p>Use LVM to create PVs and a volume group on the 22TB LUNs, something like (from memory, please check man pages):</p> <p>pvcreate /dev/sdb /dev/sdd<br/> vgcreate -n vgtest /dev/sdb /dev/sdd</p> <p>Create a 32TB LV that is using only the first 16TB of these two LUNs:</p> <p>lvcreate -n lvtest_lo -L 16T /dev/vgtest /dev/sdb<br/> lvextend -L +16T /dev/vgtest/lvtest_lo</p> <p>Create a 12TB LV that is using only the last 6TB of these two LUNs (the size may need to massaged to consume the rest of the space on both /dev/sdb and /dev/sdd:</p> <p>lvcreate -n lvtest_hi -L 6T /dev/vgtest /dev/sdb<br/> lvextend -L +6T /dev/vgtest/lvtest_hi</p> <p>What this will do is create the "lvtest_lo" on only storage space that is using blocks of the NetApp that are exclusively below 16TB, but the ldiskfs filesystem is larger than 16TB. Conversely, "lvtest_hi" is smaller than 16TB, but is using blocks of the NetApp above the 16TB limit.</p> <p>Run the original Lustre IOR test against all 4 LUNs. If "lvtest_lo" hits the problem, then it is positively caused by Lustre or ldiskfs or ext4. If "lvetst_hi" hits the problem, then it is positively a problem in the NetApp, because ldiskfs is smaller than 16TB (which didn't hit any failure before). If it hits on /dev/sda or /dev/sdc then we are confused (it could be either again), but I hope not.</p> <p>Here's the new setup for OSS, will report. <br/> &#8212; Logical volume &#8212;<br/> LV Name /dev/vglu1015/lv1015_lo<br/> VG Name vglu1015<br/> LV UUID sgpofs-dO8q-nGd1-yNlj-Vgah-L1NT-cph17Y<br/> LV Write Access read/write<br/> LV Status available</p> <ol> <li>open 0<br/> LV Size 32.00 TiB<br/> Current LE 8388608<br/> Segments 2<br/> Allocation inherit<br/> Read ahead sectors auto</li> </ol> <ul class="alternate" type="square"> <li>currently set to 256<br/> Block device 253:0</li> </ul> <p> &#8212; Segments &#8212;<br/> Logical extent 0 to 4194303:<br/> Type linear<br/> Physical volume /dev/sdb<br/> Physical extents 0 to 4194303</p> <p> Logical extent 4194304 to 8388607:<br/> Type linear<br/> Physical volume /dev/sdd<br/> Physical extents 0 to 4194303</p> <p> &#8212; Logical volume &#8212;<br/> LV Name /dev/vglu1015/lv1015_hi<br/> VG Name vglu1015<br/> LV UUID qTWfgz-fr1M-Gss2-0WmE-EJfH-0is2-bTVKGZ<br/> LV Write Access read/write<br/> LV Status available</p> <ol> <li>open 0<br/> LV Size 11.60 TiB<br/> Current LE 3040872<br/> Segments 2<br/> Allocation inherit<br/> Read ahead sectors auto</li> </ol> <ul class="alternate" type="square"> <li>currently set to 256<br/> Block device 253:1</li> </ul> <p> &#8212; Segments &#8212;<br/> Logical extent 0 to 1520435:<br/> Type linear<br/> Physical volume /dev/sdb<br/> Physical extents 4194304 to 5714739</p> <p> Logical extent 1520436 to 3040871:<br/> Type linear<br/> Physical volume /dev/sdd<br/> Physical extents 4194304 to 5714739</p> <p>I am not certain this is going to uncover errors, as the performance through LVM is about 1/10 of the native performance. Any ideas for tuning this setup for better numbers?</p> <p>That is probably Sur to seeking between the LUNs and the LVs stripes across them. I didn't think the test took so long to run, just to start testing and then reboot. </p> <p>If it needs to be faster there are less "good" tests that could be run. </p> <p>For example, run on only the hi or lo LVs at one time, alternating, and then see which one fails. That would take twice as long, but not 10x as long. We would need to run several times to be confident only one config is failing. </p> <p>The current issue is that none of the configs are failing, I was running longer in hopes of generating a failure. But a failure has not occurred since last tuesday on any configuration, so I am currently quite puzzled. There have been hardware changes durning this, is it possible that this was a hardware issue and has been fixed by the LLNL controller changes? </p> <p>To clarify, my concern is not the length of the tests, rather with 1/10 the IO rate, my concern is that we are no longer driving the hardware very hard, if this issue is related to speed or volume of IO at the device level we won't be able to replicate the issue. </p> <p>Running on <em>either</em> the hi or lo LVs at one time should get the IO rate to the one LUN back to the original level. If that does not return the symptoms again, then we need to go back to the full LUN testing without LVM to verify that the problem can still be hit. </p> <p>Returned to simpler setup, using straight ldiskfs was able to re-create errors on &gt;21TB OST. Data attached. <br/> Will try now with local IOR/ext4 to see if problem can be isolated to ldiskfs code. </p> <p>I have reformatted with ext4, running iOR locally, and have had one failure, results attached.</p> <p>Cliff, over the weekend there was a posting on the linux-ext4 list with an e2fsck patch that may resolve this problem. It seems that the root of the problem is in e2fsck itself, not ldiskfs or ext4, but is only seen if there are blocks in the journal to be recovered beyond 16TB, which is why it didn't show up regularly in testing. </p> <p>The posted patch is larger, since it also fixes some further 64-bit block number problems on 32-bit systems, but the gist of the patch is below.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">From 3b693d0b03569795d04920a04a0a21e5f64ffedc Mon Sep 17 00:00:00 2001 From: Theodore Ts'o &lt;tytso@mit.edu&gt; Date: Mon, 21 May 2012 21:30:45 -0400 Subject: [PATCH] e2fsck: fix 64-bit journal support 64-bit journal support was broken; we weren't using the high bits from the journal descriptor blocks in some cases! Signed-off-by: <span class="code-quote">"Theodore Ts'o"</span> &lt;tytso@mit.edu&gt; --- e2fsck/jfs_user.h | 4 ++-- e2fsck/journal.c | 33 +++++++++++++++++---------------- e2fsck/recovery.c | 25 ++++++++++++------------- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/e2fsck/jfs_user.h b/e2fsck/jfs_user.h index 9e33306..92f8ae2 100644 --- a/e2fsck/jfs_user.h +++ b/e2fsck/jfs_user.h @@ -18,7 +18,7 @@ struct buffer_head { e2fsck_t b_ctx; io_channel b_io; <span class="code-object">int</span> b_size; - blk_t b_blocknr; + unsigned <span class="code-object">long</span> <span class="code-object">long</span> b_blocknr; <span class="code-object">int</span> b_dirty; <span class="code-object">int</span> b_uptodate; <span class="code-object">int</span> b_err; diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c index b669941..e94ef4e 100644 --- a/e2fsck/recovery.c +++ b/e2fsck/recovery.c @@ -309,7 +309,6 @@ <span class="code-object">int</span> journal_skip_recovery(journal_t *journal) <span class="code-keyword">return</span> err; } -#<span class="code-keyword">if</span> 0 <span class="code-keyword">static</span> inline unsigned <span class="code-object">long</span> <span class="code-object">long</span> read_tag_block(<span class="code-object">int</span> tag_bytes, journal_block_tag_t *tag) { unsigned <span class="code-object">long</span> <span class="code-object">long</span> block = be32_to_cpu(tag-&gt;t_blocknr); @@ -317,7 +316,6 @@ <span class="code-keyword">static</span> inline unsigned <span class="code-object">long</span> <span class="code-object">long</span> read_tag_block(<span class="code-object">int</span> tag_bytes, journal_block_tag block |= (__u64)be32_to_cpu(tag-&gt;t_blocknr_high) &lt;&lt; 32; <span class="code-keyword">return</span> block; } -#endif /* * calc_chksums calculates the checksums <span class="code-keyword">for</span> the blocks described in the * descriptor block. @@ -506,7 +504,8 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> do_one_pass(journal_t *journal, unsigned <span class="code-object">long</span> blocknr; J_ASSERT(obh != NULL); - blocknr = be32_to_cpu(tag-&gt;t_blocknr); + blocknr = read_tag_block(tag_bytes, + tag); /* If the block has been * revoked, then we're all done </pre> </div></div> <p>Bumping priority on this for tracking. It is a bug in e2fsprogs, not Lustre, but making it a blocker ensures it will get continuous attention.</p> <p>e2fsprogs-1.42.3.wc1 (tag v1.42.3.wc1 in git) has been built and packages are available for testing:</p> <p><a href="http://build.whamcloud.com/job/e2fsprogs-master/arch=x86_64,distro=el6/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/e2fsprogs-master/arch=x86_64,distro=el6/</a></p> <p>Cliff, could you please give this a test (even better to run it in a loop) and see if it resolves the problem?</p> <p>Running with latest e2fsprogs, one error recovered, logs attached.</p> <p>/dev/vglu1015/lv1015_hi: catastrophic mode - not reading inode or group bitmaps<br/> lu1015-OST0000: recovering journal<br/> lu1015-OST0000: Truncating orphaned inode 78643270 (uid=0, gid=0, mode=0100666, size=0)<br/> lu1015-OST0000: Inode 78643270, i_size is 0, should be 16777216. FIXED.<br/> lu1015-OST0000: 119/182453760 files (1.7% non-contiguous), 57592447/3113852928 blocks</p> <p>file is lu1015.060812.tar.gz on the FTP site</p> <p>Cliff, how many runs did it take to hit this error?</p> <p>I don't think this is related to the problem seen before. Truncating orphan inodes on recovery is normal behaviour when a file is in the middle of being truncated at crash time. It looks like this handling isn't tested very often and has a bug because the "Truncating orphaned inode" message means the inode should be truncated to size=0 bytes, but then e2fsck gets confused and detects the file size is smaller than the allocated blocks and resets the size to cover the allocated blocks. This should be filed &amp; fixed separately.</p> <p>The error occured on the second run. The system ran large-lun.sh successfully prior to this.</p> <p>I've been able to reproduce this bug in vanilla e2fsck, and the problem exists only for large extent-mapped files that are being truncated at the time of a crash. </p> <p>Problem is fixed in released e2fsprogs-1.42.3.wc1.</p> Development Epic metadata Rank 1|hzuru7: Rank (Obsolete) 2172 Severity