Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.5.3
-
None
-
CentOS-6.7
lustre-2.5.3
lvm2-2.02.118-3.el6_7.4
Also note that the MDT uses an external journal device.
-
3
-
9223372036854775807
Description
Similar to LU-7616 "creation of LVM snapshot on ldiskfs based MDT hangs until MDT activity/use is halted", but opening a new case for tracking.
The goal is to use LVM snapshots and tar to make file level MDT backups. Procedure worked fine 2 or 3 times, then we triggered the following problem on a recent attempt.
The MDS became extremely sluggish, and all MDT threads went into D state, when running the following command:
lvcreate -l95%FREE -s -p r -n mdt_snap /dev/nbp9-vg/mdt9
(the command never returned, and any further lv* commands hung as well)
In the logs...
Apr 25 17:09:35 nbp9-mds kernel: WARNING: at /usr/src/redhat/BUILD/lustre-2.5.3/ldiskfs/super.c:280 ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs]() (Not tainted) Apr 25 17:14:45 nbp9-mds ] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa0e5c4e5>] ? mds_readpage_handle+0x15/0x20 [mdt] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa08a90c5>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa05d18d5>] ? lc_watchdog_touch+0x65/0x170 [libcfs] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa08a1a69>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa08ab89d>] ? ptlrpc_main+0xafd/0x1780 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffff8100c28a>] ? child_rip+0xa/0x20 Apr 25 17:14:45 nbp9-mds kernel: [<ffffffffa08aada0>] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20 Apr 25 17:14:45 nbp9-mds kernel: ---[ end trace c9f3339c0e103edf ]--- Apr 25 17:14:57 nbp9-mds kernel: WARNING: at /usr/src/redhat/BUILD/lustre-2.5.3/ldiskfs/super.c:280 ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs]() (Tainted: G W --------------- ) Apr 25 17:14:57 nbp9-mds kernel: Hardware name: AltixXE270 Apr 25 17:14:57 nbp9-mds kernel: Modules linked in: dm_snapshot dm_bufio osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) jbd2 lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic crc32c_intel libcfs(U) sunrpc bonding ib_ucm(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) configfs ib_ipoib(U) ib_cm(U) ib_uverbs(U) ib_umad(U) dm_round_robin scsi_dh_rdac dm_multipath microcode iTCO_wdt iTCO_vendor_support i2c_i801 lpc_ich mfd_core shpchp sg igb dca ptp pps_core tcp_bic ext3 jbd sd_mod crc_t10dif sr_mod cdrom ahci pata_acpi ata_generic pata_jmicron mptfc scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6 mlx4_core(U) mlx_compat(U) memtrack(U) usb_storage radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod gru [last unloaded: scsi_wait_scan] Apr 25 17:14:57 nbp9-mds kernel: Pid: 85906, comm: mdt_rdpg00_042 Tainted: G W --------------- 2.6.32-504.30.3.el6.20151008.x86_64.lustre253 #1 Apr 25 17:14:57 nbp9-mds kernel: Call Trace: Apr 25 17:14:57 nbp9-mds kernel: [<ffffffff81074127>] ? warn_slowpath_common+0x87/0xc0 Apr 25 17:14:57 nbp9-mds kernel: [<ffffffff8107417a>] ? warn_slowpath_null+0x1a/0x20 Apr 25 17:14:57 nbp9-mds kernel: [<ffffffffa0a1c33e>] ? ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs] Apr 25 17:14:57 nbp9-mds kernel: [<ffffffffa0d6069f>] ? osd_trans_start+0x1df/0x660 [osd_ldiskfs] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0ef3619>] ? lod_trans_start+0x1b9/0x250 [lod] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0f7af07>] ? mdd_trans_start+0x17/0x20 [mdd] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0f61ece>] ? mdd_close+0x6be/0xb80 [mdd] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0e48be9>] ? mdt_mfd_close+0x4a9/0x1bc0 [mdt] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0899525>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa08c07f6>] ? __req_capsule_get+0x166/0x710 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa089a53e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa06ebf05>] ? class_handle2object+0x95/0x190 [obdclass] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0e4b6a2>] ? mdt_close+0x642/0xa80 [mdt] Apr 25 17:15:06 nbp9-mds kernel: [<ffffffffa0e1fada>] ? mdt_handle_common+0x52a/0x1470 [mdt] Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: sdc - rdac checker reports path is down Apr 25 17:15:10 nbp9-mds multipathd: checker failed path 8:32 in map nbp9_MGS_MDS Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: remaining active paths: 1 Apr 25 17:15:10 nbp9-mds multipathd: sdd: remove path (uevent) Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: failed in domap for removal of path sdd Apr 25 17:15:10 nbp9-mds multipathd: uevent trigger error Apr 25 17:15:10 nbp9-mds multipathd: sdc: remove path (uevent) Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa0e5c4e5>] ? mds_readpage_handle+0x15/0x20 [mdt] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa08a90c5>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa05d18d5>] ? lc_watchdog_touch+0x65/0x170 [libcfs] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa08a1a69>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa08ab89d>] ? ptlrpc_main+0xafd/0x1780 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffff8100c28a>] ? child_rip+0xa/0x20 Apr 25 17:15:10 nbp9-mds kernel: [<ffffffffa08aada0>] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20 Apr 25 17:15:16 nbp9-mds multipathd: nbp9_MGS_MDS: map in use Apr 25 17:15:16 nbp9-mds multipathd: nbp9_MGS_MDS: can't flush
The server had to be rebooted and e2fsck run to get it back into production.
Attachments
Issue Links
- is related to
-
-
- Open
-
Activity
Full listing of nonzero files in lost+found, gathered as:
cd /nobackupp9/.lustre/lost+found/MDT0000 find . -type f -a ! -size 0 | xargs ls -ld > /u/ndauchy/tmp/nbp9_diag/lostfound_nonzero.lst
The closest match I could find in lost+found was:
[0x20011c02e:0x1e13:0x0]-[0x20001d596:0x97:0x0]-3-C-0 [0x20011c02e:0x69c7:0x0]-[0x20001d596:0x97:0x0]-0-C-0 [0x20011c030:0x1219c:0x0]-[0x20001d596:0x97:0x0]-2-C-0 [0x20011c030:0x13ffc:0x0]-[0x20001d596:0x97:0x0]-1-C-0
And the file ownership is the same as /nobackupp9/dtalcott/nagtest.toobig
To follow up on question from yesterday, which you probably already guessed...
There are 240 OSTs on the corrupted file system (on 16 OSS).
By default, the repairing behaviour will be recorded in Lustre debug log via label "D_LFSCK". But because Lustre kernel log is in RAM only, and if you did not dump them periodically, then it will be overwritten. I am not sure whether you have such log or not.
On the other hand, the log "nagtest.toobig.stripes" shows that there are 244 OST-objects claiming as the stripe of the [0x20001d596:0x97:0x0], but the "lfs getstripe" output shows that the file "nagtest.toobig" only has 240 stripes. So there are (at least) 4 OST-objects are fake. I found that there are 4 OST-objects with the size 251658240 bytes are conflict with another 4 OST-objects with the size 4194304 bytes, as shown following:
A) 65.out.service162:F,2833,10491,1179,362692,251658240,lma,[0x100000000:0x93bfc2:0x0],0,fid,[0x20001d596:0x97:0x0],0,1461812807,1461812805,1461812807 B) 104.out.service169:F,240,10491,1179,30814,251658240,lma,[0x100000000:0x9e78e3:0x0],0,fid,[0x20001d596:0x97:0x1],1,1461812807,1461812805,1461812807 C) 121.out.service170:F,2375,10491,1179,304121,251658240,lma,[0x100000000:0x969e66:0x0],0,fid,[0x20001d596:0x97:0x2],2,1461812807,1461812805,1461812807 D) 87.out.service168:F,4361,10491,1179,558333,251658240,lma,[0x100000000:0x97b343:0x0],0,fid,[0x20001d596:0x97:0x3],3,1461812807,1461812805,1461812807 a) 56.out.service169:F,32528,10491,1179,4163662,4194304,lma,[0x100000000:0x512a2:0x0],0,fid,[0x20001d596:0x97:0x0],0,1461629209,1461629204,1461629209 b) 207.out.service176:F,37845,10491,1179,4844182,4194304,lma,[0x100000000:0x5139a:0x0],0,fid,[0x20001d596:0x97:0x1],1,1461629209,1461629204,1461629209 c) 137.out.service170:F,40964,10491,1179,5243393,4194304,lma,[0x100000000:0x512d2:0x0],0,fid,[0x20001d596:0x97:0x2],2,1461629209,1461629204,1461629209 d) 236.out.service173:F,35030,10491,1179,4483887,4194304,lma,[0x100000000:0x51320:0x0],0,fid,[0x20001d596:0x97:0x3],3,1461629209,1461629204,1461629209
The OST-objects A-D with the size 251658240 bytes claim as the first 4 stripes of the file "nagtest.toobig". The reason may be as following: the file "nagtest.toobig" lost its LOV EA because some unknown reason. Either the OST-objects A-D or a-d have corrupted PFID EA. According to the size attribute, the OST-objects A-D seems have corrupted PFID EA. During the LFSCK processing, the layout LFSCK tried to re-geneate the "nagtest.toobig" LOV EA with these 244 orphan OST-objects. Unfortunately, the OST-objects A-D were found earlier than the OST-objects a-d, then the regenerated LOV EA contains the OST-objects A-D information. And then when the orphan OST-objects a-d were handled, the layout LFSCK found that they were conflict with some others, then the layout LFSCK should have created some new files with the name "[0x20001d596:0x97:0x0]-[0x100000000:0x512a2:0x0]-C-0" or similar under the directory ".lustre/lost+found/MDT0000/". Please check for verification.
Is there a way log what files that lfsck is making changes to.
As for the /nobackupp9/dtalcott/nagtest.toobig. I am attaching object scan of all osts for this file. You can see that there was 2 stripes 0. All the stripe for this file are 4194304 in size but the stripe on OST-65 is 251658240.
#filename:type,blockgroup,uid,gid,inode,size,type,fid,stripe,ctime,atime,mtime
56.out.service169:F,32528,10491,1179,4163662,4194304,lma,[0x100000000:0x512a2:0x0],0,fid,[0x20001d596:0x97:0x0],0,1461629209,1461629204,1461629209
65.out.service162:F,2833,10491,1179,362692,251658240,lma,[0x100000000:0x93bfc2:0x0],0,fid,[0x20001d596:0x97:0x0],0,1461812807,1461812805,1461812807
and ost 65 has stripe 62
#filename:type,blockgroup,uid,gid,inode,size,type,fid,stripe,ctime,atime,mtime
65.out.service162:F,39432,10491,1179,5047326,4194304,lma,[0x100000000:0x51208:0x0],0,fid,[0x20001d596:0x97:0xd8],216,1461629208,1461629204,1461629208
How many OSTs in your system? Have you ever created the file nagtest.toobig with 240 stripes? If no, then it may because that some OST-object(s) contained bad PFID EA, and claimed that its parent is the file nagtest.toobig, but LFSCK could NOT know whether it is right or not. Since nobody conflict with it/them, then the nagtest.toobig's LOV EA is enlarged to contain those OST-object(s) as to the size increased.
According to the "lfs getstripe" output, we know every OST-object's ID, then we can use such ID to check on related OST. For example:
obdidx objid objid group
65 9682882 0x93bfc2 0
Here "65" corresponding to the OST0041 (HEX), the "9682882" corresponding to the sub-dir "9682882 % 32 = 2" under /O, you can check such OST-object via: debugfs -c -R "stat /O/0/d2/9682882" $OST0041_dev
On the other hand, the OST-objects with ID "332xx" looks abnormal. Usually, the OST-object's ID on different OSTs should be quite scattered. But your case looks some adjacent numbers. That makes me to suspect that the LOV EA is over-written by some pattern data.
Note: we do eventually still want to get to the bottom of the snapshot journal call trace and hang issue.
But first, another possible corruption scenario... A file used for hourly I/O tests grew from 960M before the lfsck to 57G afterward. It appears the extra size is mostly zeros, embedded in the middle of the file.
# stat /nobackupp9/dtalcott/nagtest.toobig File: `/nobackupp9/dtalcott/nagtest.toobig' Size: 60150513664 Blocks: 3899408 IO Block: 4194304 regular file Device: 162deh/90846d Inode: 144117204932100247 Links: 1 Access: (0444/-r--r--r--) Uid: (10491/dtalcott) Gid: ( 1179/ cstaff) Access: 2016-04-29 12:36:38.000000000 -0700 Modify: 2016-04-28 14:06:34.000000000 -0700 Change: 2016-04-29 11:55:56.000000000 -0700 Birth: - # stat /nobackupp9/dtalcott/nagtest File: `/nobackupp9/dtalcott/nagtest' Size: 1006632960 Blocks: 1966080 IO Block: 4194304 regular file Device: 162deh/90846d Inode: 144134705246109726 Links: 1 Access: (0644/-rw-r--r--) Uid: (10491/dtalcott) Gid: ( 1179/ cstaff) Access: 2016-05-02 10:06:50.000000000 -0700 Modify: 2016-05-02 10:06:56.000000000 -0700 Change: 2016-05-02 10:06:56.000000000 -0700 Birth: -
(file was simply renamed, and a new one created for other testing)
# lfs getstripe /nobackupp9/dtalcott/nagtest.toobig /nobackupp9/dtalcott/nagtest.toobig lmm_stripe_count: 240 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 236 lmm_stripe_offset: 65 obdidx objid objid group 65 9682882 0x93bfc2 0 104 10385635 0x9e78e3 0 121 9870950 0x969e66 0 87 9941827 0x97b343 0 170 332727 0x513b7 0 13 332418 0x51282 0 6 332591 0x5132f 0 171 332503 0x512d7 0 33 332294 0x51206 0 21 332406 0x51276 0 195 332547 0x51303 0 126 332574 0x5131e 0 162 332560 0x51310 0 103 332341 0x51235 0 144 332517 0x512e5 0 196 332547 0x51303 0 184 332443 0x5129b 0 239 332564 0x51314 0 89 332596 0x51334 0 60 332234 0x511ca 0 26 332362 0x5124a 0 61 332282 0x511fa 0 134 332304 0x51210 0 187 332391 0x51267 0 209 332400 0x51270 0 213 333037 0x514ed 0 211 332515 0x512e3 0 30 332396 0x5126c 0 50 332564 0x51314 0 151 332187 0x5119b 0 32 332532 0x512f4 0 20 332721 0x513b1 0 72 332643 0x51363 0 143 332666 0x5137a 0 73 332940 0x5148c 0 92 332507 0x512db 0 10 332659 0x51373 0 109 332645 0x51365 0 166 332316 0x5121c 0 203 332306 0x51212 0 193 332351 0x5123f 0 149 332595 0x51333 0 3 332815 0x5140f 0 94 332773 0x513e5 0 18 332809 0x51409 0 183 332545 0x51301 0 80 332312 0x51218 0 212 332258 0x511e2 0 120 332442 0x5129a 0 191 332265 0x511e9 0 9 332443 0x5129b 0 108 332580 0x51324 0 122 332412 0x5127c 0 173 332369 0x51251 0 38 332405 0x51275 0 123 332523 0x512eb 0 1 332561 0x51311 0 5 332717 0x513ad 0 131 332752 0x513d0 0 158 332602 0x5133a 0 210 332550 0x51306 0 167 332499 0x512d3 0 64 332551 0x51307 0 116 332689 0x51391 0 24 332897 0x51461 0 111 332624 0x51350 0 169 332665 0x51379 0 220 332578 0x51322 0 186 332801 0x51401 0 205 332960 0x514a0 0 70 332644 0x51364 0 75 332327 0x51227 0 225 332395 0x5126b 0 133 332705 0x513a1 0 147 332449 0x512a1 0 46 332438 0x51296 0 114 332486 0x512c6 0 231 332310 0x51216 0 112 332828 0x5141c 0 52 332709 0x513a5 0 136 332738 0x513c2 0 95 332467 0x512b3 0 105 332314 0x5121a 0 172 332398 0x5126e 0 218 332224 0x511c0 0 189 332112 0x51150 0 86 332331 0x5122b 0 27 332303 0x5120f 0 161 332587 0x5132b 0 117 332643 0x51363 0 179 332506 0x512da 0 238 332437 0x51295 0 226 332206 0x511ae 0 23 332669 0x5137d 0 208 332492 0x512cc 0 228 332685 0x5138d 0 216 332582 0x51326 0 223 332566 0x51316 0 185 332800 0x51400 0 28 332903 0x51467 0 138 332639 0x5135f 0 157 332528 0x512f0 0 22 332328 0x51228 0 235 332309 0x51215 0 113 332504 0x512d8 0 181 332486 0x512c6 0 227 332562 0x51312 0 174 332442 0x5129a 0 178 332552 0x51308 0 55 332324 0x51224 0 48 332698 0x5139a 0 148 332589 0x5132d 0 152 332613 0x51345 0 159 332651 0x5136b 0 41 332617 0x51349 0 156 332609 0x51341 0 202 332762 0x513da 0 125 332840 0x51428 0 230 332751 0x513cf 0 43 332886 0x51456 0 145 332769 0x513e1 0 229 332760 0x513d8 0 83 332440 0x51298 0 14 332357 0x51245 0 82 332559 0x5130f 0 119 332679 0x51387 0 16 332860 0x5143c 0 180 332702 0x5139e 0 200 332568 0x51318 0 175 332710 0x513a6 0 153 332796 0x513fc 0 124 332438 0x51296 0 58 332344 0x51238 0 141 332188 0x5119c 0 182 332413 0x5127d 0 91 332378 0x5125a 0 81 332441 0x51299 0 69 332581 0x51325 0 51 332391 0x51267 0 190 332423 0x51287 0 98 332557 0x5130d 0 199 332471 0x512b7 0 160 332528 0x512f0 0 84 332693 0x51395 0 88 332739 0x513c3 0 79 332455 0x512a7 0 57 332518 0x512e6 0 44 332375 0x51257 0 90 332369 0x51251 0 45 332629 0x51355 0 214 332659 0x51373 0 107 332818 0x51412 0 49 332966 0x514a6 0 37 332933 0x51485 0 99 332822 0x51416 0 110 332478 0x512be 0 130 332654 0x5136e 0 71 332394 0x5126a 0 96 332281 0x511f9 0 4 332616 0x51348 0 104 332652 0x5136c 0 15 332590 0x5132e 0 121 332463 0x512af 0 76 332578 0x51322 0 42 332582 0x51326 0 29 332589 0x5132d 0 102 332444 0x5129c 0 219 332330 0x5122a 0 129 332467 0x512b3 0 85 332737 0x513c1 0 19 332804 0x51404 0 206 332591 0x5132f 0 34 332583 0x51327 0 7 332498 0x512d2 0 176 332351 0x5123f 0 164 332292 0x51204 0 8 332694 0x51396 0 47 332683 0x5138b 0 217 332318 0x5121e 0 188 333078 0x51516 0 74 332354 0x51242 0 77 332476 0x512bc 0 198 332637 0x5135d 0 11 332451 0x512a3 0 177 333010 0x514d2 0 101 332725 0x513b5 0 163 332432 0x51290 0 62 332414 0x5127e 0 146 332412 0x5127c 0 135 332565 0x51315 0 128 332862 0x5143e 0 100 332828 0x5141c 0 168 332484 0x512c4 0 127 332413 0x5127d 0 201 332432 0x51290 0 140 332907 0x5146b 0 154 332681 0x51389 0 221 332738 0x513c2 0 54 332759 0x513d7 0 155 332498 0x512d2 0 17 332498 0x512d2 0 165 332624 0x51350 0 67 332566 0x51316 0 142 332525 0x512ed 0 194 332587 0x5132b 0 87 332708 0x513a4 0 224 332711 0x513a7 0 132 332822 0x51416 0 232 332903 0x51467 0 63 332711 0x513a7 0 233 332720 0x513b0 0 12 332866 0x51442 0 234 332943 0x5148f 0 93 332571 0x5131b 0 118 332352 0x51240 0 59 332427 0x5128b 0 65 332296 0x51208 0 197 332254 0x511de 0 35 332315 0x5121b 0 222 332360 0x51248 0 66 332418 0x51282 0 215 332613 0x51345 0 192 332397 0x5126d 0 36 332430 0x5128e 0 40 332322 0x51222 0 31 332412 0x5127c 0 25 332424 0x51288 0 204 332576 0x51320 0 106 332455 0x512a7 0 237 332340 0x51234 0 150 332376 0x51258 0 139 332451 0x512a3 0 97 332461 0x512ad 0 53 332565 0x51315 0 115 332749 0x513cd 0 78 332803 0x51403 0 2 332885 0x51455 0 39 332810 0x5140a 0 0 333030 0x514e6 0 68 332710 0x513a6 0
So far I have not been able to figure out any reasonable way to scan for other files that grew a lot and got filled with zeros. The best I can think of is a find for all files modified since 4/28 that are more than ~1GB. Then compare the alleged size (ls -l) and the blocks used (du -s), and if they are more than ~10x off it could be a problem, and look at those files individually. Ugh! Any other ideas?
For the files under .lustre/lost+found/MDT0000/, handle them as following:
1) If the file is accessible (means no IO error), and its size is NOT zero, then it contains some valid data. But since we do not know its original position in the namespace, you needs to move them back to the namespace according to your memory or some new rule.
2) If the file is accessible, and if the size is zero, then it is safe to remove them directly, at least it does not contains valid data, although it may affect related inode quota a bit.
3) If the file is inaccessible with IO error, means only part of the stripes are found, and there is 'hole' in the re-generated LOV EA. Then you can dump part of the data from such file via "dd conv=sync,noerror ...". If such part data is still valuable, then save them in some new file. Otherwise, you have to discard them.
There are indeed about 464350 files in the .lustre/lost+found/MDT0000/ directory. The vast majority of them are owned by root, size zero, and date stamped around when the lfsck was run... so I'm assuming those are useless.
In going through the remaining ~969 "interesting" ones, several have IO errors:
pfe27 /nobackupp9/.lustre/lost+found/MDT0000 # file [0x2000a1e14:0x10e4b:0x0]-R-0 [0x2000a1e14:0x10e4b:0x0]-R-0: ERROR: cannot read `' (Input/output error) pfe27 /nobackupp9/.lustre/lost+found/MDT0000 # ls -l [0x2000a1e14:0x10e4b:0x0]-R-0 -r-------- 1 dlrodrig g28100 20971520 Apr 28 20:31 [0x2000a1e14:0x10e4b:0x0]-R-0 pfe27 /nobackupp9/.lustre/lost+found/MDT0000 # lfs getstripe !$ lfs getstripe [0x2000a1e14:0x10e4b:0x0]-R-0 [0x2000a1e14:0x10e4b:0x0]-R-0 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 40000001 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 0 0 0 0 0 0 0 0 0 0 0 23 7103687 0x6c64c7 0
Would I be correct in assuming that those are non-recoverable?
There are several possible cases for the remained 125 no-stripes files.
1) The file has never been written after create, so the OST-object has no parent FID information.
2) The OST-object's parent FID EA is corrupted, so the OST-object does not know which file it belongs to. Then the layout LFSCK will mark it as Lustre orphan, and put under the directory "$mount_point/.lustre/lost+found/MDT0000/".
3) The OST-object was lost also.
4) There was something wrong during the layout LFSCK as to the layout LFSCK failed to re-generate the LOV EA.
According to my experience, the first case more possible.
The layout LFSCK can recover file owner information, but cannot recover ACL. Sorry for such inconvenience.
There are 3 kinds of files under your lustre lost+found directory. They are:
1) The file name contains infix "-C-". That means multiple OST-objects claim the same MDT-object and the same slot in the LOV EA. Then the layout LFSCK will create new MDT-object(s) to hold the conflict OST-object(s). Searching your "lostfound_nonzero.lst" file, there are 4 files with "-C-" name. As described in former comment, they are corresponding to the file "nagtest.toobig.stripes" and conflict with the OST-objects A-D. If you think that the layout LFSCK made wrong decision when re-generated the "nagtest.toobig.stripes" LOV EA, we need to make new patch to recover it. But since there is only one file with conflict OST-objects, you also can recover the file "nagtest.toobig.stripes" manually.
2) The file name contains infix "-R-". That means the orphan OST-object knows its parent MDT-object FID, but does not know the position (the file name) in the namespace. So have to create a file with the known PFID. Under such case, if you can remember something from file data, you can rename the file back to the normal namespace.
3) The file name contains infix "-O-". They are orphan MDT-objects. The backend e2fsck put them into the local /lost+found directory, that is invisible to the client. The LFSCK scans the MDT-objects under local /lost+found directory, if the MDT-object has valid linkEA, then the LFSCK will move it back to normal namespace, otherwise, it will be put under the global Lustre lost+found directory that is visible to the client. Similar as above case 2), if you can remember something from file data, you can rename the file back to the normal namespace. On the other hand, I noticed that the files with "-O-" name all have the size 5792, that is mysterious. Unless the e2fsck found some corrupted inodes, otherwise, it is almost impossible. So you have to check these file with more human knowledge and if you think they are invalid, then remove them directly.