[LU-6414] du and df disagree for used space Created: 31/Mar/15 Updated: 11/Jun/20 Resolved: 28/Sep/15 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.7.0, Lustre 2.5.3 |
| Fix Version/s: | None |
| Type: | Question/Request | Priority: | Minor |
| Reporter: | Frederik Ferner (Inactive) | Assignee: | nasf (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
debug_message-LU-6414.tar.gz
lustre_debug_LU-6414.tar.bz2
|
||||||||
| Issue Links: |
|
||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
On one of our file systems we have a large discrepancy between the output of du for the whole file system and df (lfs df and standard df report the same numbers), du is reporting about 120TB less disk usage than df (on a ~430TB file system, with 30 OSTs).
As the file system is currently ~80% full, this difference is of quite some concern to us. My initial suspicion is that there might be some objects allocated on the OSTs which are not reference by files, so don't show up in du, but with Lustre 2.5, I'm not entirely sure anymore how do check this. If I remember correctly at least on 1.8 I would have to run (the e2fsck based) lfsck, ideally with the file system offline/idle, which from memory was fairly invasive. I've tried to read up on the new lfsck but it's not immediately obvious to me a) if this is the right tool and b) how to invoke it to achieve my goal of freeing the ~120TB additional space. |
| Comments |
| Comment by Peter Jones [ 01/Apr/15 ] |
|
Bobijam will assist with this issue |
| Comment by Zhenyu Xu [ 02/Apr/15 ] |
|
The lfsck in 2.6 contains OST orphan object cleanup feature while unfortunately 2.5 does not have it. And in 2.5 the old way (running e2fsck based lfsck) needs to bring the file system offline. |
| Comment by Frederik Ferner (Inactive) [ 02/Apr/15 ] |
|
Thanks, this confirms how I understood the documentation. I assume that unfortunately it will not be sufficient to have a 2.6 client to run the new lfsck on? (All clients and servers are currently on 2.5.3). Can you confirm this? |
| Comment by Zhenyu Xu [ 02/Apr/15 ] |
|
You are right, servers also need to be 2.6 to support orphan objects cleanup feature of new lfsck. |
| Comment by Frederik Ferner (Inactive) [ 02/Apr/15 ] |
|
Thanks for confirming. In this case, and as I seem to remember the offline (e2fsck based) lfsck is a rather long process, we'll have to wait until the next maintenance window to do anything about this. At that point we might consider upgrading to 2.6 (or later). Thanks, |
| Comment by Frederik Ferner (Inactive) [ 03/Jun/15 ] |
|
The file systems have been upgrade to Lustre 2.7 now. I've started to look into the lfsck options for online lfsck. I was going to start the online lfsck like this, hoping to run a dry run without modifying anything first: lctl lfsck_start -M play01-MDT0000 -n on --type layout I was expecting to see a "param dryrun" in the lctl get_param -n mdd.play01-MDT0000.lfsck_layout output, but there wasn't. Could you confirm if I'm on the right track? Should the dry-run option show up in the get_param output? |
| Comment by Frederik Ferner (Inactive) [ 18/Jun/15 ] |
|
Coming back to this problem, looks like I need to find better documentation on lfsck, pointers welcome, all I really found so far was the current Lustre manual, which explains the options but I'm not sure I understand all the implications right. After upgrading the production file system to 2.7 and running lfsck namespace and layout check, as far as I can see the amount of free space reported by df hasn't changed (size reported by du hasn't increased either.). For reference I ran the following command and as far as I can see (get_param output), it has completed... lctl lfsck_start -M lustre03-MDT0000 --type layout |
| Comment by Frederik Ferner (Inactive) [ 28/Jul/15 ] |
|
Any suggestions how we can identify there the additional space (as reported by df) is used and ideally reclaim it? |
| Comment by nasf (Inactive) [ 30/Jul/15 ] |
|
Would you please to check whether there are files under $MNT/.lustre/lost+found/? They are the orphans found by the layout LFSCK. If you think the orphans are still useful, you can move them back to normal namespace (via normal mv command); otherwise, you can remove them (via normal rm command), then the space occupied by the orphans will be freed. But if there is no orphan found by the layout LFSCK, then please show me the result on MDS "lctl get_param mdd.lustre03-MDT0000.lfsck_layout". Thanks! |
| Comment by Frederik Ferner (Inactive) [ 30/Jul/15 ] |
|
There is no .lustre directory for that file system on our clients, I assume there is no special mount option we need to use to access it? [bnh65367@cs04r-sc-mds03-01 ~]$ lctl get_param mdd.lustre03-MDT0000.lfsck_layout mdd.lustre03-MDT0000.lfsck_layout= name: lfsck_layout magic: 0xb1734d76 version: 2 status: completed flags: param: time_since_last_completed: 3674064 seconds time_since_latest_start: 3687541 seconds time_since_last_checkpoint: 3674064 seconds latest_start_position: 12 last_checkpoint_position: 507503435 first_failure_position: 399704724 success_count: 1 repaired_dangling: 0 repaired_unmatched_pair: 8 repaired_multiple_referenced: 0 repaired_orphan: 0 repaired_inconsistent_owner: 54823284 repaired_others: 35760261 skipped: 0 failed_phase1: 8345 failed_phase2: 0 checked_phase1: 104823145 checked_phase2: 0 run_time_phase1: 13478 seconds run_time_phase2: 0 seconds average_speed_phase1: 7777 items/sec average_speed_phase2: 0 objs/sec real-time_speed_phase1: N/A real-time_speed_phase2: N/A current_position: N/A [bnh65367@cs04r-sc-mds03-01 ~]$ |
| Comment by nasf (Inactive) [ 30/Jul/15 ] |
|
".lustre" is a hidden directory under the mount point of the Lustre system, it will be auto created when the MDT is mount up. Would you please to show me the output of "ls -ail $MNT" ? Thanks! |
| Comment by Frederik Ferner (Inactive) [ 30/Jul/15 ] |
|
la -ail on one of our clients: (MNT=/mnt/lustre03 mount point for the file system in question...) [bnh65367@cs04r-sc-com11-01 ~]$ ls -ail $MNT
total 216
244318209 drwxr-xr-x 18 root root 4096 Jun 10 10:35 .
1048579 drwxr-xr-x 6 root root 4096 Apr 28 16:12 ..
244332133 -rw-rw-r-- 1 kdf51254 kdf51254 15 Apr 4 2012 .hello
245016855 dr-xr-xr-x 4 root root 4096 Oct 7 2014 i02
244318210 dr-xr-xr-x 12 root root 4096 Nov 17 2010 i03
244721811 dr-xr-xr-x 4 root root 4096 Sep 8 2014 i04
245016864 dr-xr-xr-x 4 root root 4096 Nov 8 2011 i04-1
144115947194548259 drwxr-xr-x 4 root root 4096 Jan 9 2015 i14
144115947194548264 drwxr-xr-x 4 root root 4096 Jan 9 2015 i14-1
144116079214460929 dr-xr-xr-x 4 root i18_data 4096 Sep 16 2014 i18
244332079 dr-xr-xr-x 4 root root 4096 May 16 2013 i23
245016873 dr-xr-xr-x 4 root root 4096 Sep 1 2014 i24
244332698 drwxr-xr-x 4 root root 4096 Jun 27 2014 m01
144116229555093633 drwxr-xr-x 4 root root 4096 Jun 10 10:36 m02
144116229555093634 drwxr-xr-x 4 root root 4096 Jun 10 10:36 m03
244404425 drwxrwxr-x+ 98 root dls_staff 12288 Jul 24 15:44 mx-scratch
244318219 drwsrwsr-x+ 19 bnh65367 dls_sysadmin 139264 Jul 21 11:37 testdir
244998177 drwxr-xr-x 4 root root 4096 Jan 28 2014 x01
[bnh65367@cs04r-sc-com11-01 ~]$
|
| Comment by Frederik Ferner (Inactive) [ 30/Jul/15 ] |
|
Just remember, if it makes any difference, this file system had initially be installed and formated using lustre 1.8 (or maybe even lustre 1.6, I'd have to check our records to be sure.) |
| Comment by nasf (Inactive) [ 01/Aug/15 ] |
|
Frederik, Would you please to umount the MDT, and mount it as "ldiskfs", then "ls -ail $MNT/ROOT/". If the ".lustre" is still NOT there, then please mount the MDT as "lustre" again with -1 level debug enabled and collect the log during the MDT mount. Thanks! |
| Comment by Frederik Ferner (Inactive) [ 03/Aug/15 ] |
|
Ok, I will look into scheduling a downtime for this, however it might not be possible for a week or two, I'll let you know. In the mean time, I have checked all our other file systems, including our pre-production file system and none of them have a .lustre directory. Would it be worth starting this investigation on our pre-prod file system, where a suitable maintenance window is much easier to arrange? |
| Comment by nasf (Inactive) [ 03/Aug/15 ] |
|
".lustre" is introduced since Lustre-2.0. For the system that was created as Lustre-1.8 or Lustre-1.6, when it is upgraded to Lustre-2.x, it is expected that the ".lustre" will be generated automatically. So if you can describe your system upgrading history, then perhaps we can re-prodouce the issues locally. |
| Comment by Frederik Ferner (Inactive) [ 04/Aug/15 ] |
|
(ah, looks like my comment yesterday didn't get posted The file system where this started has been created as lustre 1.8, updated to 2.5 and now 2.7, I'd have to check the exact versions if you need them. The pre-prod file system as well as the second production file system both have been initially created as lustre 1.6, upgraded to lustre 1.8, 2.5 and now 2.7, with the pre-prod file system briefly running on 2.3 as well (before 2.5). |
| Comment by nasf (Inactive) [ 05/Aug/15 ] |
|
Frederik, What is the output for "ls -ailR /mnt/lustre03/.lustre" on the client? |
| Comment by Frederik Ferner (Inactive) [ 05/Aug/15 ] |
|
Ah, there is something (but no files): [bnh65367@cs04r-sc-com11-01 ~]$ sudo !! sudo ls -ailR /mnt/lustre03/.lustre /mnt/lustre03/.lustre: total 8 144115188109410306 d--x------ 2 root root 4096 Aug 19 2014 fid 144115188109410307 dr-x------ 3 root root 4096 Jun 1 12:07 lost+found /mnt/lustre03/.lustre/fid: total 4 144115188109410306 d--x------ 2 root root 4096 Aug 19 2014 . /mnt/lustre03/.lustre/lost+found: total 8 144115188109410307 dr-x------ 3 root root 4096 Jun 1 12:07 . 144116228800118785 drwx------ 3 root root 4096 Jun 1 18:54 MDT0000 /mnt/lustre03/.lustre/lost+found/MDT0000: total 8 144116228800118785 drwx------ 3 root root 4096 Jun 1 18:54 . 144115188109410307 dr-x------ 3 root root 4096 Jun 1 12:07 .. [bnh65367@cs04r-sc-com11-01 ~]$ |
| Comment by nasf (Inactive) [ 05/Aug/15 ] |
|
It means that no orphan OST-objects have been found during the layout LFSCK. According to the lfsck_layout output, it may be related with "failed_phase1: 8345". But without the detailed layout LFSCK debug logs, we do not know what happened. Another possible reason for the invisible space is that there are some in-using orphans, means that some clients may still open the orphans and cause that the orphans (their MDT-objects) have not been destroyed, then the layout LFSCK cannot find related orphan OST-objects. |
| Comment by Frederik Ferner (Inactive) [ 06/Aug/15 ] |
I've been trying to find this, how would we generate the LFSCK debug logs?
As far as I can see it is unlikely that clients still have the files open, considering how much data we are talking about and that all clients have been rebooted since we've opened the ticket. |
| Comment by nasf (Inactive) [ 06/Aug/15 ] |
lctl set_param debug="+lfsck" If all the clients have been rebooted, then the open-unlinked orphans should have been destroyed. |
| Comment by nasf (Inactive) [ 06/Aug/15 ] |
|
We need the lfsck debug logs on both MDS and OSS. |
| Comment by Frederik Ferner (Inactive) [ 06/Aug/15 ] |
|
sorry, should have asked before, what is the expected performance impact on general file system performance if we run the layout lfsck with full debug enabled on the MDS and all OSS nodes? I expect there is going to be a hit, but wonder if it is something we could do during quite times in normal operation or if we need to schedule a maintenance/at risk period. |
| Comment by nasf (Inactive) [ 06/Aug/15 ] |
|
According to our former tests, with full speed layout LFSCK run at background, the worst performance overhead will be less than 30%. Please note that we needs lfsck debug log (debug="+lfsck"), not the full lustre debug log (debug=-1). The later case will impact the performance much. It is not required to run layout LFSCK during maintenance period, instead, you can run layout LFSCK at background with normal Lustre serves the clients. To be clear, my understand is that you have rebooted all the clients, then after the client remount, you compared the "du -xk" and "lfs df -h" results, and then found quite difference, right? |
| Comment by Frederik Ferner (Inactive) [ 06/Aug/15 ] |
Correct, the exact numbers have changed but the difference is still the same order of magnitude as before. And too be certain, could you suggest the best command to start the layout LFSCK background check? I'm assuming the command I posted above (lctl lfsck_start -M lustre03-MDT0000 --type layout) is correct, but could you confirm? |
| Comment by nasf (Inactive) [ 06/Aug/15 ] |
|
lctl lfsck_start -M lustre03-MDT0000 -t layout -r -o |
| Comment by Frederik Ferner (Inactive) [ 10/Aug/15 ] |
|
I've managed to run the lfsck again with lfsck debug on. [bnh65367@cs04r-sc-mds03-01 log]$ cat /proc/fs/lustre/mdd/lustre03-MDT0000/lfsck_layout name: lfsck_layout magic: 0xb1734d76 version: 2 status: completed flags: param: all_targets,orphan time_since_last_completed: 6246 seconds time_since_latest_start: 20473 seconds time_since_last_checkpoint: 6246 seconds latest_start_position: 12 last_checkpoint_position: 507503435 first_failure_position: 399704698 success_count: 2 repaired_dangling: 0 repaired_unmatched_pair: 0 repaired_multiple_referenced: 0 repaired_orphan: 0 repaired_inconsistent_owner: 68156343 repaired_others: 0 skipped: 0 failed_phase1: 8345 failed_phase2: 0 checked_phase1: 134842146 checked_phase2: 19 run_time_phase1: 14217 seconds run_time_phase2: 10 seconds average_speed_phase1: 9484 items/sec average_speed_phase2: 1 objs/sec real-time_speed_phase1: N/A real-time_speed_phase2: N/A current_position: N/A [bnh65367@cs04r-sc-mds03-01 log]$ cat /proc/sys/lnet/debug ioctl neterror warning error emerg ha config console lfsck [bnh65367@cs04r-sc-mds03-01 log]$ There are still no files in the .lustre/lost+found directory on a client and df still reports the same usage. I'll attach /var/log/messages for all servers involved to this call. |
| Comment by nasf (Inactive) [ 11/Aug/15 ] |
|
Sorry, I think I did not explain clearly. The attached logs are syslog/dmesg, NOT Lustre debug log. Lustre debug log can be collected via "lctl dk > $log". On the other hand, the attachment contains a lot of page allocation error that caused some layout lfsck and other threads failures. |
| Comment by Frederik Ferner (Inactive) [ 11/Aug/15 ] |
|
Ah, sorry. I suspect the interesting parts of the lustre debug log has probably been overwritten by now due to the normal file system operations, so should I collect the logs now or should I run another lfsck and collect the logs there? (I'll upload the debug logs I've collected just now in any case.) The page allocation failures are only on the MDS, correct? I'm tempted to attempt to free some memory by dropping all caches (on the Lustre clients and on the MDS) before trying again, does this sound like it might be more likely to succeed? |
| Comment by nasf (Inactive) [ 11/Aug/15 ] |
|
You have to re-run the lfsck (lctl lfsck_start -M lustre03-MDT0000 -t layout -r -o) to collecting the Lustre debug logs. As for the page allocation error, it seems not lfsck special, but caused layout lfsck failure. Bobi, would you please to give some investigation about the page allocation error? Thanks! |
| Comment by Frederik Ferner (Inactive) [ 11/Aug/15 ] |
|
Is it worth to try and increase the debug buffer (and maybe even use a debug_daemon) to capture more of the logs? (Not sure how much information you are expecting and if debug_mb=241 is big enough...) |
| Comment by nasf (Inactive) [ 11/Aug/15 ] |
|
I am also not sure how much information will be generated, depends on the device size and inconsistency status. Anyway, according to my local test, 241 MB is not quite large buffer for that, more large buffer is better. |
| Comment by Frederik Ferner (Inactive) [ 12/Aug/15 ] |
|
Ok, I ran another lfsck last night with debug_daemon enabled (10GB on the MDS, 1G on each OSS) after dropping all caches on the MDS. Initially this looked better and didn't immediately cause memory allocation failures, though later during the run it started to show the same memory allocation failures but fewer, see below for a log extract, let me know if you need more details. Aug 11 17:29:30 cs04r-sc-mds03-01 kernel: lfsck: page allocation failure. order:2, mode:0x20 Aug 11 17:29:31 cs04r-sc-mds03-01 kernel: ptlrpcd_30: page allocation failure. order:2, mode:0x20 Aug 11 17:33:20 cs04r-sc-mds03-01 kernel: ptlrpcd_18: page allocation failure. order:2, mode:0x20 Aug 11 18:14:08 cs04r-sc-mds03-01 kernel: ptlrpcd_24: page allocation failure. order:2, mode:0x20 Aug 11 19:06:56 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20 Aug 11 19:15:55 cs04r-sc-mds03-01 kernel: swapper: page allocation failure. order:2, mode:0x20 This lfsck run completed as well and I have lustre debug logs. As they exceed the file upload limits for jira, I'm currently uploading them to a ftp server on our side and will provide the link once they are available. (The files should now be available at ftp://ftpanon.diamond.ac.uk/LU-6414/ with one file per server.) I have confirmed that they seem to include lfsck information and also all seem to cover the whole time it ran at least on the OSSes, though I'm not 100% sure on the later. I noticed at least on the MDS there have been two occasions where some of the debug information has been dropped (memory pressure?) Aug 11 16:46:53 cs04r-sc-mds03-01 kernel: Lustre: debug daemon will attempt to start writing to /exports/lfsck_debug_daemon_2 (10240000kB max) Aug 11 20:08:02 cs04r-sc-mds03-01 kernel: debug daemon buffer overflowed; discarding 10% of pages (103 of 1024) Aug 11 20:08:32 cs04r-sc-mds03-01 kernel: debug daemon buffer overflowed; discarding 10% of pages (103 of 1024) And just to confirm, no new files in /mnt/lustre03/.lustre anywhere. |
| Comment by Frederik Ferner (Inactive) [ 13/Aug/15 ] |
|
Apologies if it appears I'm pushing on this, but have you had a chance to download the debug logs I provided? If there is any chance that we need to take the file system offline, our best chance would be some time next week, but we'd have to tell our users as much in advance as possible. Kind regards, |
| Comment by nasf (Inactive) [ 13/Aug/15 ] |
|
If your system is still online, would you please to show me the output on MDS "lctl get_param -n mdd.lustre03-MDT0000.lfsck_async_windows" ? Thanks! |
| Comment by Frederik Ferner (Inactive) [ 14/Aug/15 ] |
[bnh65367@cs04r-sc-mds03-01 exports]$ lctl get_param -n mdd.lustre03-MDT0000.lfsck_async_windows 1024 [bnh65367@cs04r-sc-mds03-01 exports]$ |
| Comment by Frederik Ferner (Inactive) [ 14/Aug/15 ] |
|
Quick question. As it seems we're not having much luck finding the data using the online LFSCK, what else could we do during a maintenance window where the file system can be taken offline? Is the old offline lfsck likely to produce better results? What else could/should we try? |
| Comment by Andreas Dilger [ 14/Aug/15 ] |
|
Frederik, it is not possible to safely run the old offline lfsck on newer filesystems. It does not understand all of the newer features and could cause corruption in some cases. If you are having problems with debug_daemon consuming too much memory, another option is to use lctl set_param printk=+lfsck which will print the LFSCK messages to the console log (and presumably to /var/log/messages) but this may be quite a large number of messages. At least it would not consume kernel memory, just space on disk as it is logged (though it may slow down the server if there are too many messages). Fan Yong, is there some way to check if LFSCK is skipping the OST orphan recovery stage? The only other things I can think of at this point is to check at this point is if there are large open files, or otherwise writing a script that checks OST objects and then verifies that the corresponding MDT inodes still exist. To check the number of open files: lctl get_param -n mdt.*.exports.*.open_files | wc -l If you want to get the pathnames of open files, you need to mount the client filesystem on the MDS temporarily and run: lctl get_param -n mdt.*.exports.*.open_files | while read FID; do lfs fid2path $MOUNT "$FID" done At that point it is also possible to check for large files and such, but it isn't clear that is worthwhile unless there are unexpectedly large numbers of open files. |
| Comment by Joseph Gmitter (Inactive) [ 14/Aug/15 ] |
|
Assigning to Fan Yong to investigate it further per Andreas' comment. |
| Comment by nasf (Inactive) [ 14/Aug/15 ] |
|
According to the log lustre_log_cs04r-sc-oss03-04.log, the layout LFSCK found some candidate orphan OST-objects during the layout LFSCK as following: 00100000:10000000:18.0:1439212742.173670:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848711:0x0], PFID [0x200010b0d:0xf724:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.173697:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848712:0x0], PFID [0x200010b0d:0xf72d:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.173719:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848714:0x0], PFID [0x200010b0d:0xf76c:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.173813:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484871d:0x0], PFID [0x200010b0d:0xf814:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.173855:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848722:0x0], PFID [0x200010b0d:0xf866:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.173940:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484872c:0x0], PFID [0x200010b0d:0xf90a:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174017:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848735:0x0], PFID [0x200010bd0:0xbb80:0x0], owner 12798:12798 00100000:10000000:18.0:1439212742.174084:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484873d:0x0], PFID [0x200010b0d:0xfa02:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174110:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848740:0x0], PFID [0x200010b0d:0xfa44:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174138:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848743:0x0], PFID [0x200010b0d:0xfa54:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174175:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848747:0x0], PFID [0x200010b0d:0xfa83:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174219:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484874c:0x0], PFID [0x200010b0d:0xfadd:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174309:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848755:0x0], PFID [0x200010b0d:0xfb82:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174362:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484875b:0x0], PFID [0x200010b0d:0xfbc6:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174379:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484875d:0x0], PFID [0x200010b0d:0xfc0b:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174459:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848765:0x0], PFID [0x200010b0d:0xfc66:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174575:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x484876f:0x0], PFID [0x200010b0d:0xfcf4:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174620:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848774:0x0], PFID [0x200010b0d:0xfd46:0x0], owner 37641:112827 00100000:10000000:18.0:1439212742.174665:0:47611:0:(lfsck_layout.c:6084:lfsck_orphan_it_next()) lustre03-OST0017-osd: return orphan [0x100170000:0x4848779:0x0], PFID [0x200010b0d:0xfd87:0x0], owner 37641:112827 We say "candidate" orphan, because there were some memory related failures on the MDS and may cause the layout LFSCK to miss to handle some cases. These candidate orphan OST-objects have been returned by the layout LFSCK to the MDS for further verification. Unfortunately, the lustre_log_cs04r-sc-mds03-01.log does not contains any LFSCK logs. It seems that the layout LFSCK logs on the MDS has been over-written. So I cannot say what happened on the MDS. So please do the following: 1) On the client, check whether the specified parent MDT-objects exist or not by "lfs fid2path $MNT $PFID". The "MNT" is the Lustre mount point, the PFID is the string after the "PFID" in about logs, such as [0x200010b0d:0xf724:0x0]. 2) If the specified parent MDT-objects do not exist, then the candidate orphan OST-objects are real orphan OST-objects. Otherwise if some parent MDT-object is there, assume its name is $NAME, then run "lfs getstripe $NAME" on the client, that will tell you whether the MDT-object recognize the specified "orphan" OST-object or not. If yes, then it is not real orphan. Before we can resolve the memory related failure on the MDS, we have to make such manually verification. I am sorry for such inconvenience. If we can find that there are really some unless orphan OST-objects, then we can removed them with offline mode during your maintenance windows. |
| Comment by nasf (Inactive) [ 15/Aug/15 ] |
|
Frederik, If the MDS has NOT been restarted after the layout LFSCK (with memory allocation errors), then please show me the output of "cat /proc/slabinfo" on the MDS. Thanks! |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
|
Thanks all for the updates, I'll work through the suggestions ASAP.
The MDS has not been restarted after the layout LFSCK and I don't think we've dropped caches either, so here is /proc/slabinfo: [bnh65367@cs04r-sc-mds03-01 exports]$ cat /proc/slabinfo slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> fuse_request 0 0 424 9 1 : tunables 54 27 8 : slabdata 0 0 0 fuse_inode 0 0 768 5 1 : tunables 54 27 8 : slabdata 0 0 0 osp_obj 748 5111 208 19 1 : tunables 120 60 8 : slabdata 269 269 0 mdd_obj 37610 148800 80 48 1 : tunables 120 60 8 : slabdata 3100 3100 0 lod_obj 37585 131712 120 32 1 : tunables 120 60 8 : slabdata 4116 4116 0 mdt_cdt_agent_req 0 0 168 23 1 : tunables 120 60 8 : slabdata 0 0 0 mdt_cdt_restore_handle 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 mdt_obj 37556 103008 248 16 1 : tunables 120 60 8 : slabdata 6438 6438 0 osd_itea_cache 19 27 416 9 1 : tunables 54 27 8 : slabdata 3 3 0 dynlock_cache 135 265 72 53 1 : tunables 120 60 8 : slabdata 5 5 0 upd_kmem 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0 lqe_kmem 4 19 200 19 1 : tunables 120 60 8 : slabdata 1 1 0 ldiskfs_inode_cache 5124921 6522342 1040 3 1 : tunables 24 12 8 : slabdata 2174114 2174114 0 ldiskfs_xattr 0 0 88 44 1 : tunables 120 60 8 : slabdata 0 0 0 ldiskfs_free_data 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 ldiskfs_alloc_context 40 84 136 28 1 : tunables 120 60 8 : slabdata 3 3 0 ldiskfs_prealloc_space 32 340 112 34 1 : tunables 120 60 8 : slabdata 10 10 0 ldiskfs_system_zone 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 xattr_kmem 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 ccc_thread_kmem 163 171 408 9 1 : tunables 54 27 8 : slabdata 19 19 0 vvp_session_kmem 34 60 256 15 1 : tunables 120 60 8 : slabdata 4 4 0 vvp_thread_kmem 163 180 376 10 1 : tunables 54 27 8 : slabdata 18 18 0 vvp_req_kmem 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 vvp_object_kmem 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 vvp_lock_kmem 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 ll_rmtperm_hash_cache 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 ll_remote_perm_cache 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 ll_file_data 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 lustre_inode_cache 0 0 1216 3 1 : tunables 24 12 8 : slabdata 0 0 0 lov_oinfo 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 lov_lock_link_kmem 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 lovsub_req_kmem 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 lovsub_object_kmem 0 0 168 23 1 : tunables 120 60 8 : slabdata 0 0 0 lovsub_lock_kmem 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 lov_req_kmem 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 lov_session_kmem 34 40 392 10 1 : tunables 54 27 8 : slabdata 4 4 0 lov_thread_kmem 163 176 352 11 1 : tunables 54 27 8 : slabdata 16 16 0 lov_object_kmem 0 0 232 17 1 : tunables 120 60 8 : slabdata 0 0 0 lov_lock_kmem 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 interval_node 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 ldlm_locks 75337 96313 512 7 1 : tunables 54 27 8 : slabdata 13759 13759 0 ldlm_resources 39856 74544 320 12 1 : tunables 54 27 8 : slabdata 6212 6212 13 ptlrpc_cache 1052 1240 768 5 1 : tunables 54 27 8 : slabdata 247 248 34 cl_env_kmem 1 23 168 23 1 : tunables 120 60 8 : slabdata 1 1 0 capa_cache 0 0 184 21 1 : tunables 120 60 8 : slabdata 0 0 0 ll_import_cache 0 0 1440 5 2 : tunables 24 12 8 : slabdata 0 0 0 ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 ll_obd_dev_cache 41 46 3528 2 2 : tunables 24 12 8 : slabdata 23 23 0 fat_inode_cache 0 0 672 6 1 : tunables 54 27 8 : slabdata 0 0 0 fat_cache 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 nfs_direct_cache 0 0 200 19 1 : tunables 120 60 8 : slabdata 0 0 0 nfs_commit_data 0 0 704 11 2 : tunables 54 27 8 : slabdata 0 0 0 nfs_write_data 36 36 960 4 1 : tunables 54 27 8 : slabdata 9 9 0 nfs_read_data 0 0 896 4 1 : tunables 54 27 8 : slabdata 0 0 0 nfs_inode_cache 46 48 1048 3 1 : tunables 24 12 8 : slabdata 16 16 0 nfs_page 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 fscache_cookie_jar 8 96 80 48 1 : tunables 120 60 8 : slabdata 2 2 0 rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 : slabdata 4 4 0 rpc_tasks 8 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 rpc_inode_cache 30 32 832 4 1 : tunables 54 27 8 : slabdata 8 8 0 jbd2_1k 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 dm_mpath_io 320 606 16 202 1 : tunables 120 60 8 : slabdata 3 3 0 ext4_inode_cache 88996 89048 1000 4 1 : tunables 54 27 8 : slabdata 22262 22262 0 ext4_xattr 1 44 88 44 1 : tunables 120 60 8 : slabdata 1 1 0 ext4_free_block_extents 0 0 56 67 1 : tunables 120 60 8 : slabdata 0 0 0 ext4_alloc_context 16 28 136 28 1 : tunables 120 60 8 : slabdata 1 1 0 ext4_prealloc_space 83 148 104 37 1 : tunables 120 60 8 : slabdata 4 4 0 ext4_system_zone 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 jbd2_journal_handle 158 432 24 144 1 : tunables 120 60 8 : slabdata 3 3 0 jbd2_journal_head 621 952 112 34 1 : tunables 120 60 8 : slabdata 28 28 0 jbd2_revoke_table 10 202 16 202 1 : tunables 120 60 8 : slabdata 1 1 0 jbd2_revoke_record 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 sd_ext_cdb 2 112 32 112 1 : tunables 120 60 8 : slabdata 1 1 0 ib_mad 2114 2135 512 7 1 : tunables 54 27 8 : slabdata 305 305 0 fib6_nodes 26 118 64 59 1 : tunables 120 60 8 : slabdata 2 2 0 ip6_dst_cache 24 50 384 10 1 : tunables 54 27 8 : slabdata 5 5 0 ndisc_cache 3 30 256 15 1 : tunables 120 60 8 : slabdata 2 2 0 ip6_mrt_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 RAWv6 51 52 1024 4 1 : tunables 54 27 8 : slabdata 13 13 0 UDPLITEv6 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 UDPv6 10 12 1024 4 1 : tunables 54 27 8 : slabdata 3 3 0 tw_sock_TCPv6 0 0 320 12 1 : tunables 54 27 8 : slabdata 0 0 0 request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 TCPv6 7 8 1920 2 1 : tunables 24 12 8 : slabdata 4 4 0 scsi_tgt_cmd 0 0 80 48 1 : tunables 120 60 8 : slabdata 0 0 0 scsi_sense_cache 77 120 128 30 1 : tunables 120 60 8 : slabdata 4 4 0 scsi_cmd_cache 72 90 256 15 1 : tunables 120 60 8 : slabdata 6 6 0 dm_raid1_read_record 0 0 1064 7 2 : tunables 24 12 8 : slabdata 0 0 0 kcopyd_job 0 0 3240 2 2 : tunables 24 12 8 : slabdata 0 0 0 io 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 dm_uevent 0 0 2608 3 2 : tunables 24 12 8 : slabdata 0 0 0 dm_rq_clone_bio_info 391 808 16 202 1 : tunables 120 60 8 : slabdata 4 4 0 dm_rq_target_io 282 290 392 10 1 : tunables 54 27 8 : slabdata 29 29 0 dm_target_io 218 864 24 144 1 : tunables 120 60 8 : slabdata 6 6 0 dm_io 217 644 40 92 1 : tunables 120 60 8 : slabdata 7 7 0 flow_cache 0 0 104 37 1 : tunables 120 60 8 : slabdata 0 0 0 uhci_urb_priv 0 0 56 67 1 : tunables 120 60 8 : slabdata 0 0 0 cfq_io_context 0 0 136 28 1 : tunables 120 60 8 : slabdata 0 0 0 cfq_queue 0 0 240 16 1 : tunables 120 60 8 : slabdata 0 0 0 bsg_cmd 0 0 312 12 1 : tunables 54 27 8 : slabdata 0 0 0 mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 : slabdata 1 1 0 isofs_inode_cache 0 0 640 6 1 : tunables 54 27 8 : slabdata 0 0 0 hugetlbfs_inode_cache 1 6 608 6 1 : tunables 54 27 8 : slabdata 1 1 0 dquot 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 kioctx 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 inotify_event_private_data 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 inotify_inode_mark_entry 118 224 120 32 1 : tunables 120 60 8 : slabdata 7 7 0 dnotify_mark_entry 0 0 120 32 1 : tunables 120 60 8 : slabdata 0 0 0 dnotify_struct 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 dio 0 0 640 6 1 : tunables 54 27 8 : slabdata 0 0 0 fasync_cache 1 144 24 144 1 : tunables 120 60 8 : slabdata 1 1 0 khugepaged_mm_slot 17 368 40 92 1 : tunables 120 60 8 : slabdata 4 4 0 ksm_mm_slot 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 ksm_stable_node 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 ksm_rmap_item 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 utrace_engine 0 0 56 67 1 : tunables 120 60 8 : slabdata 0 0 0 utrace 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 pid_namespace 0 0 2168 3 2 : tunables 24 12 8 : slabdata 0 0 0 posix_timers_cache 0 0 176 22 1 : tunables 120 60 8 : slabdata 0 0 0 uid_cache 9 60 128 30 1 : tunables 120 60 8 : slabdata 2 2 0 UNIX 165 288 832 9 2 : tunables 54 27 8 : slabdata 32 32 0 ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 UDP-Lite 0 0 896 4 1 : tunables 54 27 8 : slabdata 0 0 0 tcp_bind_bucket 26 177 64 59 1 : tunables 120 60 8 : slabdata 3 3 0 inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0 secpath_cache 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 xfrm_dst_cache 0 0 448 8 1 : tunables 54 27 8 : slabdata 0 0 0 ip_fib_alias 2 224 32 112 1 : tunables 120 60 8 : slabdata 2 2 0 ip_fib_hash 19 106 72 53 1 : tunables 120 60 8 : slabdata 2 2 0 ip_dst_cache 765 1540 384 10 1 : tunables 54 27 8 : slabdata 154 154 0 arp_cache 11 30 256 15 1 : tunables 120 60 8 : slabdata 2 2 0 PING 0 0 832 9 2 : tunables 54 27 8 : slabdata 0 0 0 RAW 49 54 832 9 2 : tunables 54 27 8 : slabdata 6 6 0 UDP 26 36 896 4 1 : tunables 54 27 8 : slabdata 9 9 0 tw_sock_TCP 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 TCP 1104 1160 1728 4 2 : tunables 24 12 8 : slabdata 290 290 0 eventpoll_pwq 57 159 72 53 1 : tunables 120 60 8 : slabdata 3 3 0 eventpoll_epi 57 180 128 30 1 : tunables 120 60 8 : slabdata 6 6 0 sgpool-256 2 2 8192 1 2 : tunables 8 4 0 : slabdata 2 2 0 sgpool-128 2 2 4096 1 1 : tunables 24 12 8 : slabdata 2 2 0 sgpool-64 3 4 2048 2 1 : tunables 24 12 8 : slabdata 2 2 0 sgpool-32 5 8 1024 4 1 : tunables 54 27 8 : slabdata 2 2 0 sgpool-16 2 16 512 8 1 : tunables 54 27 8 : slabdata 2 2 0 sgpool-8 71 105 256 15 1 : tunables 120 60 8 : slabdata 7 7 0 scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 blkdev_queue 36 38 2864 2 2 : tunables 24 12 8 : slabdata 19 19 0 blkdev_requests 104 132 352 11 1 : tunables 54 27 8 : slabdata 12 12 0 blkdev_ioc 222 288 80 48 1 : tunables 120 60 8 : slabdata 6 6 0 fsnotify_event_holder 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 fsnotify_event 0 0 104 37 1 : tunables 120 60 8 : slabdata 0 0 0 bio-0 641 1060 192 20 1 : tunables 120 60 8 : slabdata 52 53 128 biovec-256 338 338 4096 1 1 : tunables 24 12 8 : slabdata 338 338 0 biovec-128 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0 biovec-64 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 biovec-16 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 bip-256 2 2 4224 1 2 : tunables 8 4 0 : slabdata 2 2 0 bip-128 0 0 2176 3 2 : tunables 24 12 8 : slabdata 0 0 0 bip-64 0 0 1152 7 2 : tunables 24 12 8 : slabdata 0 0 0 bip-16 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 bip-4 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 bip-1 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 sock_inode_cache 1438 1580 704 5 1 : tunables 54 27 8 : slabdata 316 316 0 skbuff_fclone_cache 341 364 512 7 1 : tunables 54 27 8 : slabdata 52 52 58 skbuff_head_cache 1688 2100 256 15 1 : tunables 120 60 8 : slabdata 140 140 323 file_lock_cache 56 66 176 22 1 : tunables 120 60 8 : slabdata 3 3 0 net_namespace 0 0 2432 3 2 : tunables 24 12 8 : slabdata 0 0 0 shmem_inode_cache 1064 1185 784 5 1 : tunables 54 27 8 : slabdata 237 237 0 Acpi-Operand 1222 1749 72 53 1 : tunables 120 60 8 : slabdata 33 33 0 Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-State 0 0 80 48 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-Namespace 985 1104 40 92 1 : tunables 120 60 8 : slabdata 12 12 0 task_delay_info 1583 2210 112 34 1 : tunables 120 60 8 : slabdata 65 65 0 taskstats 7 24 328 12 1 : tunables 54 27 8 : slabdata 2 2 0 proc_inode_cache 14318 14634 656 6 1 : tunables 54 27 8 : slabdata 2439 2439 0 sigqueue 123 168 160 24 1 : tunables 120 60 8 : slabdata 7 7 0 bdev_cache 36 68 832 4 1 : tunables 54 27 8 : slabdata 17 17 0 sysfs_dir_cache 27368 27432 144 27 1 : tunables 120 60 8 : slabdata 1016 1016 0 mnt_cache 40 120 256 15 1 : tunables 120 60 8 : slabdata 8 8 0 filp 2690 5400 192 20 1 : tunables 120 60 8 : slabdata 270 270 384 inode_cache 6219 6354 592 6 1 : tunables 54 27 8 : slabdata 1059 1059 0 dentry 95450 97540 192 20 1 : tunables 120 60 8 : slabdata 4877 4877 248 names_cache 61 61 4096 1 1 : tunables 24 12 8 : slabdata 61 61 0 avc_node 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 selinux_inode_security 9473 10070 72 53 1 : tunables 120 60 8 : slabdata 190 190 0 radix_tree_node 832993 837872 560 7 1 : tunables 54 27 8 : slabdata 119696 119696 0 key_jar 10 40 192 20 1 : tunables 120 60 8 : slabdata 2 2 0 buffer_head 26042516 28841315 104 37 1 : tunables 120 60 8 : slabdata 779495 779495 0 nsproxy 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 vm_area_struct 5763 6783 200 19 1 : tunables 120 60 8 : slabdata 357 357 156 mm_struct 104 185 1408 5 2 : tunables 24 12 8 : slabdata 37 37 0 fs_cache 692 1121 64 59 1 : tunables 120 60 8 : slabdata 19 19 0 files_cache 155 308 704 11 2 : tunables 54 27 8 : slabdata 28 28 0 signal_cache 1423 1477 1088 7 2 : tunables 24 12 8 : slabdata 211 211 0 sighand_cache 1412 1422 2112 3 2 : tunables 24 12 8 : slabdata 474 474 0 task_xstate 216 540 832 9 2 : tunables 54 27 8 : slabdata 60 60 0 task_struct 1521 1527 2656 3 2 : tunables 24 12 8 : slabdata 509 509 0 cred_jar 1857 2480 192 20 1 : tunables 120 60 8 : slabdata 124 124 0 anon_vma_chain 4955 8932 48 77 1 : tunables 120 60 8 : slabdata 116 116 236 anon_vma 3469 5244 40 92 1 : tunables 120 60 8 : slabdata 57 57 72 pid 1622 2100 128 30 1 : tunables 120 60 8 : slabdata 70 70 0 shared_policy_node 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 numa_policy 31 56 136 28 1 : tunables 120 60 8 : slabdata 2 2 0 idr_layer_cache 368 385 544 7 1 : tunables 54 27 8 : slabdata 55 55 0 size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 0 : slabdata 0 0 0 size-4194304 0 0 4194304 1 1024 : tunables 1 1 0 : slabdata 0 0 0 size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 0 : slabdata 0 0 0 size-2097152 0 0 2097152 1 512 : tunables 1 1 0 : slabdata 0 0 0 size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 0 : slabdata 0 0 0 size-1048576 1 1 1048576 1 256 : tunables 1 1 0 : slabdata 1 1 0 size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 : slabdata 0 0 0 size-524288 1 1 524288 1 128 : tunables 1 1 0 : slabdata 1 1 0 size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 : slabdata 0 0 0 size-262144 1 1 262144 1 64 : tunables 1 1 0 : slabdata 1 1 0 size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-131072 3 3 131072 1 32 : tunables 8 4 0 : slabdata 3 3 0 size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0 size-65536 85 85 65536 1 16 : tunables 8 4 0 : slabdata 85 85 0 size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-32768 200 200 32768 1 8 : tunables 8 4 0 : slabdata 200 200 0 size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0 size-16384 5983 6064 16384 1 4 : tunables 8 4 0 : slabdata 5983 6064 0 size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 size-8192 27814 27814 8192 1 2 : tunables 8 4 0 : slabdata 27814 27814 0 size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0 size-4096 3601 3714 4096 1 1 : tunables 24 12 8 : slabdata 3601 3714 0 size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0 size-2048 4954 5068 2048 2 1 : tunables 24 12 8 : slabdata 2534 2534 0 size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 size-1024 30977 31480 1024 4 1 : tunables 54 27 8 : slabdata 7870 7870 88 size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0 size-512 346008 385008 512 8 1 : tunables 54 27 8 : slabdata 48126 48126 161 size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 size-256 5234 6615 256 15 1 : tunables 120 60 8 : slabdata 441 441 0 size-192(DMA) 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 size-192 75709 203900 192 20 1 : tunables 120 60 8 : slabdata 10195 10195 0 size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 size-64 137707 212459 64 59 1 : tunables 120 60 8 : slabdata 3601 3601 0 size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 size-128 4740524 6493230 128 30 1 : tunables 120 60 8 : slabdata 216441 216441 184 size-32 67110 270928 32 112 1 : tunables 120 60 8 : slabdata 2419 2419 81 kmem_cache 245 245 32896 1 16 : tunables 8 4 0 : slabdata 245 245 0 [bnh65367@cs04r-sc-mds03-01 exports]$ |
| Comment by nasf (Inactive) [ 17/Aug/15 ] |
|
The slab used by ldiskfs_inode_cache and buffer_head looks abnormal. Would you please to show me the output on MDS "lctl get_param -n osd-ldiskfs.lustre03-MDT0000.oi_scrub" ? Thanks! |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
[bnh65367@cs04r-sc-mds03-01 exports]$ lctl get_param -n osd-ldiskfs.lustre03-MDT0000.oi_scrub name: OI_scrub magic: 0x4c5fd252 oi_files: 64 status: completed flags: param: time_since_last_completed: 493355 seconds time_since_latest_start: 506551 seconds time_since_last_checkpoint: 493355 seconds latest_start_position: 12 last_checkpoint_position: 507510785 first_failure_position: N/A checked: 134422033 updated: 0 failed: 0 prior_updated: 0 noscrub: 4577 igif: 41096443 success_count: 5 run_time: 13195 seconds average_speed: 10187 objects/sec real-time_speed: N/A current_position: N/A lf_scanned: 0 lf_reparied: 0 lf_failed: 0 [bnh65367@cs04r-sc-mds03-01 exports]$ |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
I've just checked all the candidates and for all of the them there is a MDT-object and AFAICT the object id on the OST is a part of the file for each of them. |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
|
Andreas, there are currently ~2800 files open on the MDS, I've checked them all and they are all fairly recent files as well as relatively small, the largest is 23GB, second largest 6GB. |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
|
If the list above really is the complete list of candidates for orphans then this is nowhere near enough to explain the 120TB difference between df and du. There are only 19 files involved,. as far as I can see they are all very recent and certainly <1GB in total. Is there something else that could explain this? |
| Comment by nasf (Inactive) [ 17/Aug/15 ] |
|
They are only the known candidate list from the given logs. But not all the layout LFSCK logs have been recorded. I do not know why some logs are missed, as to I cannot estimate how many orphans in the system. But according to the output from proc interface layout_lfsck on the MDS, the layout LFSCK did not find real orphans (repaired_orphan: 0). It may be because of the memory failures on the MDS, some orphans checking have been skipped. That can explain why some of the layout LFSCK logs on the OST missed. Please show me the output of "lctl get_param -n obdfilter.*.lfsck_layout" on all the OSS nodes. |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
[bnh65367@ws104 bnh65367]$ dsh -g lustre03_oss -w -- /usr/sbin/lctl get_param -n obdfilter.*.lfsck_layout cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 111761409 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3860680 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 225 seconds cs04r-sc-oss03-01: run_time_phase2: 12978 seconds cs04r-sc-oss03-01: average_speed_phase1: 17158 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 112216066 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3894604 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 225 seconds cs04r-sc-oss03-01: run_time_phase2: 12978 seconds cs04r-sc-oss03-01: average_speed_phase1: 17309 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 110998527 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3811155 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 223 seconds cs04r-sc-oss03-01: run_time_phase2: 12981 seconds cs04r-sc-oss03-01: average_speed_phase1: 17090 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 95311874 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3810965 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 209 seconds cs04r-sc-oss03-01: run_time_phase2: 12994 seconds cs04r-sc-oss03-01: average_speed_phase1: 18234 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 90354687 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3837762 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 198 seconds cs04r-sc-oss03-01: run_time_phase2: 13006 seconds cs04r-sc-oss03-01: average_speed_phase1: 19382 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 95421440 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3842114 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 223 seconds cs04r-sc-oss03-01: run_time_phase2: 12980 seconds cs04r-sc-oss03-01: average_speed_phase1: 17229 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-01: name: lfsck_layout cs04r-sc-oss03-01: magic: 0xb1734d76 cs04r-sc-oss03-01: version: 2 cs04r-sc-oss03-01: status: completed cs04r-sc-oss03-01: flags: cs04r-sc-oss03-01: param: all_targets,orphan cs04r-sc-oss03-01: time_since_last_completed: 509632 seconds cs04r-sc-oss03-01: time_since_latest_start: 522836 seconds cs04r-sc-oss03-01: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-01: latest_start_position: 12 cs04r-sc-oss03-01: last_checkpoint_position: 81060865 cs04r-sc-oss03-01: first_failure_position: 0 cs04r-sc-oss03-01: success_count: 3 cs04r-sc-oss03-01: repaired_dangling: 0 cs04r-sc-oss03-01: repaired_unmatched_pair: 0 cs04r-sc-oss03-01: repaired_multiple_referenced: 0 cs04r-sc-oss03-01: repaired_orphan: 0 cs04r-sc-oss03-01: repaired_inconsistent_owner: 0 cs04r-sc-oss03-01: repaired_others: 0 cs04r-sc-oss03-01: skipped: 0 cs04r-sc-oss03-01: failed_phase1: 0 cs04r-sc-oss03-01: failed_phase2: 0 cs04r-sc-oss03-01: checked_phase1: 3889361 cs04r-sc-oss03-01: checked_phase2: 0 cs04r-sc-oss03-01: run_time_phase1: 224 seconds cs04r-sc-oss03-01: run_time_phase2: 12980 seconds cs04r-sc-oss03-01: average_speed_phase1: 17363 items/sec cs04r-sc-oss03-01: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-01: real-time_speed_phase1: N/A cs04r-sc-oss03-01: real-time_speed_phase2: N/A cs04r-sc-oss03-01: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 108646402 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3807089 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 231 seconds cs04r-sc-oss03-02: run_time_phase2: 12973 seconds cs04r-sc-oss03-02: average_speed_phase1: 16480 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 100199421 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3829166 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 249 seconds cs04r-sc-oss03-02: run_time_phase2: 12954 seconds cs04r-sc-oss03-02: average_speed_phase1: 15378 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 110349078 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3787712 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 243 seconds cs04r-sc-oss03-02: run_time_phase2: 12961 seconds cs04r-sc-oss03-02: average_speed_phase1: 15587 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 117570537 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3756423 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 248 seconds cs04r-sc-oss03-02: run_time_phase2: 12956 seconds cs04r-sc-oss03-02: average_speed_phase1: 15146 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 89275389 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3725165 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 244 seconds cs04r-sc-oss03-02: run_time_phase2: 12960 seconds cs04r-sc-oss03-02: average_speed_phase1: 15267 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 119472128 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3738528 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 216 seconds cs04r-sc-oss03-02: run_time_phase2: 12988 seconds cs04r-sc-oss03-02: average_speed_phase1: 17308 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 99655681 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3699190 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 238 seconds cs04r-sc-oss03-02: run_time_phase2: 12966 seconds cs04r-sc-oss03-02: average_speed_phase1: 15542 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-02: name: lfsck_layout cs04r-sc-oss03-02: magic: 0xb1734d76 cs04r-sc-oss03-02: version: 2 cs04r-sc-oss03-02: status: completed cs04r-sc-oss03-02: flags: cs04r-sc-oss03-02: param: all_targets,orphan cs04r-sc-oss03-02: time_since_last_completed: 509632 seconds cs04r-sc-oss03-02: time_since_latest_start: 522836 seconds cs04r-sc-oss03-02: time_since_last_checkpoint: 509632 seconds cs04r-sc-oss03-02: latest_start_position: 12 cs04r-sc-oss03-02: last_checkpoint_position: 119319212 cs04r-sc-oss03-02: first_failure_position: 0 cs04r-sc-oss03-02: success_count: 3 cs04r-sc-oss03-02: repaired_dangling: 0 cs04r-sc-oss03-02: repaired_unmatched_pair: 0 cs04r-sc-oss03-02: repaired_multiple_referenced: 0 cs04r-sc-oss03-02: repaired_orphan: 0 cs04r-sc-oss03-02: repaired_inconsistent_owner: 0 cs04r-sc-oss03-02: repaired_others: 0 cs04r-sc-oss03-02: skipped: 0 cs04r-sc-oss03-02: failed_phase1: 0 cs04r-sc-oss03-02: failed_phase2: 0 cs04r-sc-oss03-02: checked_phase1: 3820745 cs04r-sc-oss03-02: checked_phase2: 0 cs04r-sc-oss03-02: run_time_phase1: 249 seconds cs04r-sc-oss03-02: run_time_phase2: 12955 seconds cs04r-sc-oss03-02: average_speed_phase1: 15344 items/sec cs04r-sc-oss03-02: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-02: real-time_speed_phase1: N/A cs04r-sc-oss03-02: real-time_speed_phase2: N/A cs04r-sc-oss03-02: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 105683948 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3838618 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 251 seconds cs04r-sc-oss03-03: run_time_phase2: 12953 seconds cs04r-sc-oss03-03: average_speed_phase1: 15293 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 111613954 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3823837 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 248 seconds cs04r-sc-oss03-03: run_time_phase2: 12955 seconds cs04r-sc-oss03-03: average_speed_phase1: 15418 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 119013377 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3761564 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 248 seconds cs04r-sc-oss03-03: run_time_phase2: 12955 seconds cs04r-sc-oss03-03: average_speed_phase1: 15167 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 115606527 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3880238 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 251 seconds cs04r-sc-oss03-03: run_time_phase2: 12952 seconds cs04r-sc-oss03-03: average_speed_phase1: 15459 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 93549568 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3857218 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 255 seconds cs04r-sc-oss03-03: run_time_phase2: 12949 seconds cs04r-sc-oss03-03: average_speed_phase1: 15126 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 118549502 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3783075 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 217 seconds cs04r-sc-oss03-03: run_time_phase2: 12987 seconds cs04r-sc-oss03-03: average_speed_phase1: 17433 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 119262173 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3842466 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 252 seconds cs04r-sc-oss03-03: run_time_phase2: 12951 seconds cs04r-sc-oss03-03: average_speed_phase1: 15247 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-03: name: lfsck_layout cs04r-sc-oss03-03: magic: 0xb1734d76 cs04r-sc-oss03-03: version: 2 cs04r-sc-oss03-03: status: completed cs04r-sc-oss03-03: flags: cs04r-sc-oss03-03: param: all_targets,orphan cs04r-sc-oss03-03: time_since_last_completed: 509633 seconds cs04r-sc-oss03-03: time_since_latest_start: 522837 seconds cs04r-sc-oss03-03: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-03: latest_start_position: 12 cs04r-sc-oss03-03: last_checkpoint_position: 113485826 cs04r-sc-oss03-03: first_failure_position: 0 cs04r-sc-oss03-03: success_count: 3 cs04r-sc-oss03-03: repaired_dangling: 0 cs04r-sc-oss03-03: repaired_unmatched_pair: 0 cs04r-sc-oss03-03: repaired_multiple_referenced: 0 cs04r-sc-oss03-03: repaired_orphan: 0 cs04r-sc-oss03-03: repaired_inconsistent_owner: 0 cs04r-sc-oss03-03: repaired_others: 0 cs04r-sc-oss03-03: skipped: 0 cs04r-sc-oss03-03: failed_phase1: 0 cs04r-sc-oss03-03: failed_phase2: 0 cs04r-sc-oss03-03: checked_phase1: 3820425 cs04r-sc-oss03-03: checked_phase2: 0 cs04r-sc-oss03-03: run_time_phase1: 254 seconds cs04r-sc-oss03-03: run_time_phase2: 12950 seconds cs04r-sc-oss03-03: average_speed_phase1: 15041 items/sec cs04r-sc-oss03-03: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-03: real-time_speed_phase1: N/A cs04r-sc-oss03-03: real-time_speed_phase2: N/A cs04r-sc-oss03-03: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 100287488 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3779517 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 225 seconds cs04r-sc-oss03-04: run_time_phase2: 12979 seconds cs04r-sc-oss03-04: average_speed_phase1: 16797 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 116240386 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3798986 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 225 seconds cs04r-sc-oss03-04: run_time_phase2: 12979 seconds cs04r-sc-oss03-04: average_speed_phase1: 16884 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 112877569 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3777192 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 221 seconds cs04r-sc-oss03-04: run_time_phase2: 12983 seconds cs04r-sc-oss03-04: average_speed_phase1: 17091 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 111502336 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3817548 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 228 seconds cs04r-sc-oss03-04: run_time_phase2: 12976 seconds cs04r-sc-oss03-04: average_speed_phase1: 16743 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 110585858 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3810913 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 227 seconds cs04r-sc-oss03-04: run_time_phase2: 12977 seconds cs04r-sc-oss03-04: average_speed_phase1: 16788 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 119462243 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3816819 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 222 seconds cs04r-sc-oss03-04: run_time_phase2: 12981 seconds cs04r-sc-oss03-04: average_speed_phase1: 17192 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A cs04r-sc-oss03-04: name: lfsck_layout cs04r-sc-oss03-04: magic: 0xb1734d76 cs04r-sc-oss03-04: version: 2 cs04r-sc-oss03-04: status: completed cs04r-sc-oss03-04: flags: cs04r-sc-oss03-04: param: all_targets,orphan cs04r-sc-oss03-04: time_since_last_completed: 509633 seconds cs04r-sc-oss03-04: time_since_latest_start: 522837 seconds cs04r-sc-oss03-04: time_since_last_checkpoint: 509633 seconds cs04r-sc-oss03-04: latest_start_position: 12 cs04r-sc-oss03-04: last_checkpoint_position: 116584449 cs04r-sc-oss03-04: first_failure_position: 0 cs04r-sc-oss03-04: success_count: 3 cs04r-sc-oss03-04: repaired_dangling: 0 cs04r-sc-oss03-04: repaired_unmatched_pair: 0 cs04r-sc-oss03-04: repaired_multiple_referenced: 0 cs04r-sc-oss03-04: repaired_orphan: 0 cs04r-sc-oss03-04: repaired_inconsistent_owner: 0 cs04r-sc-oss03-04: repaired_others: 0 cs04r-sc-oss03-04: skipped: 0 cs04r-sc-oss03-04: failed_phase1: 0 cs04r-sc-oss03-04: failed_phase2: 0 cs04r-sc-oss03-04: checked_phase1: 3724766 cs04r-sc-oss03-04: checked_phase2: 0 cs04r-sc-oss03-04: run_time_phase1: 223 seconds cs04r-sc-oss03-04: run_time_phase2: 12981 seconds cs04r-sc-oss03-04: average_speed_phase1: 16702 items/sec cs04r-sc-oss03-04: average_speed_phase2: 0 objs/sec cs04r-sc-oss03-04: real-time_speed_phase1: N/A cs04r-sc-oss03-04: real-time_speed_phase2: N/A cs04r-sc-oss03-04: current_position: N/A |
| Comment by nasf (Inactive) [ 17/Aug/15 ] |
|
There may be other possible reason about the "du/df" difference, not orphan. I have done the following test locally: # sh llmount.sh Stopping clients: RHEL6 /mnt/lustre (opts:) Stopping clients: RHEL6 /mnt/lustre2 (opts:) Loading modules from /root/Work/Lustre/L96/lustre-release/lustre/tests/.. detected 2 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck subsystem_debug=all -lnet -lnd -pinger ../lnet/lnet/lnet options: 'accept=all' gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients RHEL6 environments Loading modules from /root/Work/Lustre/L96/lustre-release/lustre/tests/.. detected 2 online CPUs by sysfs Force libcfs to create 2 CPU partitions debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck subsystem_debug=all -lnet -lnd -pinger gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 Started lustre-OST0001 Starting client: RHEL6: -o user_xattr,flock RHEL6@tcp:/lustre /mnt/lustre Using TIMEOUT=20 seting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90 secs for update Updated after 3s: wanted 'procname_uid' got 'procname_uid' disable quota as required # cp -rf . /mnt/lustre/ # cd /mnt/lustre # sync # du -xk 12 ./test-groups 8 ./clientapi 72 ./acl 4 ./.libs 84 ./racer 340 ./.deps 32 ./mpi/.deps 248 ./mpi 16 ./cfg 52 ./rmtacl 12 ./posix 15320 . # lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 149944 18164 121416 13% /mnt/lustre[MDT:0] lustre-OST0000_UUID 187464 31104 145780 18% /mnt/lustre[OST:0] lustre-OST0001_UUID 187464 35712 141036 20% /mnt/lustre[OST:1] filesystem summary: 374928 66816 286816 19% /mnt/lustre As you can see that the "du/df" results are quite different. Because it is a new formatted system, there should be no orphans. The difference is caused by: 2) "du" result is calculated via count all client visible files one by one. Lustre uses extent space allocation algorithm, the space occupied by the files may be larger than its shown size. So the used space showed by "du" must be larger than "du -xk". As for how much the difference, it depends on the system status. Would you please to show me the following: Thanks! |
| Comment by Frederik Ferner (Inactive) [ 17/Aug/15 ] |
|
Would all this really make that much difference? Could I check the oi.16.xx file sizes somehow? (Preferably without taking the file offline? Would this be possible with read-only debugfs somehow? Anyway, running 'du -xk' will obviously take some time, I've started it now but would not expect this to complete. (BTW: is du -sxk /mnt/lustre03 sufficient or do you want more details or even one entry per directory, which is likely to be a lot of data?) In the mean time I'll provide 'lfs df' output as well as the rbh-du summary output for the file system in case this helps already. In the past I did verify that rbh-du and standard du basically agree, they'll obviously never be quite the same due to the long run time for du to go through the whole file system. [bnh65367@cs04r-sc-com09-26 ~]$ lfs df /mnt/lustre03 UUID 1K-blocks Used Available Use% Mounted on lustre03-MDT0000_UUID 1521989576 110551268 1388420692 7% /mnt/lustre03[MDT:0] lustre03-OST0000_UUID 15261133728 13161571340 1942835728 87% /mnt/lustre03[OST:0] lustre03-OST0001_UUID 15261133728 13081866876 2022384096 87% /mnt/lustre03[OST:1] lustre03-OST0002_UUID 15261133728 13188830940 1915629344 87% /mnt/lustre03[OST:2] lustre03-OST0003_UUID 15261133728 13184314756 1920136456 87% /mnt/lustre03[OST:3] lustre03-OST0004_UUID 15261133728 13107827900 1996474144 87% /mnt/lustre03[OST:4] lustre03-OST0005_UUID 15261133728 13203240208 1901248332 87% /mnt/lustre03[OST:5] lustre03-OST0006_UUID 15261133728 13050019912 2054169036 86% /mnt/lustre03[OST:6] lustre03-OST0007_UUID 15261133728 13209241708 1895258440 87% /mnt/lustre03[OST:7] lustre03-OST0008_UUID 15261133728 13148621044 1955760520 87% /mnt/lustre03[OST:8] lustre03-OST0009_UUID 15261133728 13213433312 1891074040 87% /mnt/lustre03[OST:9] lustre03-OST000a_UUID 15261133728 13271630652 1832991248 88% /mnt/lustre03[OST:10] lustre03-OST000b_UUID 15261133728 13305996164 1798692584 88% /mnt/lustre03[OST:11] lustre03-OST000c_UUID 15261133728 13272308520 1832312860 88% /mnt/lustre03[OST:12] lustre03-OST000d_UUID 15261133728 13361524776 1743267992 88% /mnt/lustre03[OST:13] lustre03-OST000e_UUID 15261133728 13184712720 1919739296 87% /mnt/lustre03[OST:14] lustre03-OST000f_UUID 15261133728 13172786384 1931642548 87% /mnt/lustre03[OST:15] lustre03-OST0010_UUID 15261133728 13185191344 1919261064 87% /mnt/lustre03[OST:16] lustre03-OST0011_UUID 15261133728 13285184180 1819463844 88% /mnt/lustre03[OST:17] lustre03-OST0012_UUID 15261133728 13051060644 2053130244 86% /mnt/lustre03[OST:18] lustre03-OST0013_UUID 15261133728 13162554756 1941853388 87% /mnt/lustre03[OST:19] lustre03-OST0014_UUID 15261133728 13291838132 1812823364 88% /mnt/lustre03[OST:20] lustre03-OST0015_UUID 15261133728 13113037544 1991274724 87% /mnt/lustre03[OST:21] lustre03-OST0016_UUID 15261133728 13202424320 1902062652 87% /mnt/lustre03[OST:22] lustre03-OST0017_UUID 15261133728 13183362604 1921085420 87% /mnt/lustre03[OST:23] lustre03-OST0018_UUID 15261133728 13226523644 1878010348 88% /mnt/lustre03[OST:24] lustre03-OST0019_UUID 15261133728 13258957628 1845639644 88% /mnt/lustre03[OST:25] lustre03-OST001a_UUID 15261133728 13209268596 1895231272 87% /mnt/lustre03[OST:26] lustre03-OST001b_UUID 15261133728 13149994124 1954390396 87% /mnt/lustre03[OST:27] lustre03-OST001c_UUID 15261133728 13178567060 1925871232 87% /mnt/lustre03[OST:28] lustre03-OST001d_UUID 15261133728 13372645168 1732174132 89% /mnt/lustre03[OST:29] filesystem summary: 457834011840 395988536956 57145888388 87% /mnt/lustre03 [bnh65367@cs04r-sc-serv-92 ~]$ rbh-du -d -f lustre03 /mnt/lustre03 Using config file '/etc/robinhood.d/tmpfs/lustre03.conf'. /mnt/lustre03 symlink count:881618, size:55759075, spc_used:2330484736 dir count:18383038, size:77926428672, spc_used:77974368256 file count:113612618, size:257183179718967, spc_used:246216372580352 sock count:1, size:0, spc_used:0 [bnh65367@cs04r-sc-serv-92 ~]$ |
| Comment by Andreas Dilger [ 17/Aug/15 ] |
|
I don't think the overhead from the filesystem metadata should be as much as the reported 100TB of missing space. There is a 400MB journal per OST, plus 256 bytes per inode (for the total inodes reported by "lfs df -i" for each OST, whether used or free). By my estimate that works out to be roughly 40GB for your 30 OSTs, or it might be as much as 400GB depending on how the OSTs were formatted, but not 120TB. |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
|
According to the "du/df" results, there is about 1/3 space disappeared. If it is only because of orphans, there should be about 1/3 (at least tens of millions) objects are orphans unless some orphans' size is quite huge. If there are really about 1/3 objects are orphans, then even though the layout LFSCK may miss to handle some orphans, it is almost impossible to miss all of them. So I suspect that there should be some other reasons although I do not know them yet. About checking the oi.16.xx size, you can use debugfs with online mode. For example: debugfs -c -R 'stat oi.16.10' $device. |
| Comment by Frederik Ferner (Inactive) [ 18/Aug/15 ] |
|
Andreas, thanks for confirming that my assumptions how much overhead to expect were basically right. FYI, 'lfs df -i' output below. Could I also just make sure that I've not missed any request for information on anything that you want me to provide? (other than maybe the standard du, which is still running). And also, if there is anything to be gained by some offline investigation, tomorrow is our best chance to schedule this for the next few months. [bnh65367@cs04r-sc-serv-92 ~]$ lfs df -i /mnt/lustre03 UUID Inodes IUsed IFree IUse% Mounted on lustre03-MDT0000_UUID 507510784 130371598 377139186 26% /mnt/lustre03[MDT:0] lustre03-OST0000_UUID 119472128 3766424 115705704 3% /mnt/lustre03[OST:0] lustre03-OST0001_UUID 119472128 3801681 115670447 3% /mnt/lustre03[OST:1] lustre03-OST0002_UUID 119472128 3720219 115751909 3% /mnt/lustre03[OST:2] lustre03-OST0003_UUID 119472128 3723299 115748829 3% /mnt/lustre03[OST:3] lustre03-OST0004_UUID 119472128 3739097 115733031 3% /mnt/lustre03[OST:4] lustre03-OST0005_UUID 119472128 3753254 115718874 3% /mnt/lustre03[OST:5] lustre03-OST0006_UUID 119472128 3791428 115680700 3% /mnt/lustre03[OST:6] lustre03-OST0007_UUID 119472128 3719595 115752533 3% /mnt/lustre03[OST:7] lustre03-OST0008_UUID 119472128 3729508 115742620 3% /mnt/lustre03[OST:8] lustre03-OST0009_UUID 119472128 3689953 115782175 3% /mnt/lustre03[OST:9] lustre03-OST000a_UUID 119472128 3672757 115799371 3% /mnt/lustre03[OST:10] lustre03-OST000b_UUID 119472128 3633239 115838889 3% /mnt/lustre03[OST:11] lustre03-OST000c_UUID 119472128 3656382 115815746 3% /mnt/lustre03[OST:12] lustre03-OST000d_UUID 119472128 3609695 115862433 3% /mnt/lustre03[OST:13] lustre03-OST000e_UUID 119472128 3732814 115739314 3% /mnt/lustre03[OST:14] lustre03-OST000f_UUID 119472128 3743798 115728330 3% /mnt/lustre03[OST:15] lustre03-OST0010_UUID 119472128 3742417 115729711 3% /mnt/lustre03[OST:16] lustre03-OST0011_UUID 119472128 3683660 115788468 3% /mnt/lustre03[OST:17] lustre03-OST0012_UUID 119472128 3783646 115688482 3% /mnt/lustre03[OST:18] lustre03-OST0013_UUID 119472128 3758072 115714056 3% /mnt/lustre03[OST:19] lustre03-OST0014_UUID 119472128 3695529 115776599 3% /mnt/lustre03[OST:20] lustre03-OST0015_UUID 119472128 3747648 115724480 3% /mnt/lustre03[OST:21] lustre03-OST0016_UUID 119472128 3739399 115732729 3% /mnt/lustre03[OST:22] lustre03-OST0017_UUID 119472128 3680837 115791291 3% /mnt/lustre03[OST:23] lustre03-OST0018_UUID 119472128 3700590 115771538 3% /mnt/lustre03[OST:24] lustre03-OST0019_UUID 119472128 3684233 115787895 3% /mnt/lustre03[OST:25] lustre03-OST001a_UUID 119472128 3717452 115754676 3% /mnt/lustre03[OST:26] lustre03-OST001b_UUID 119472128 3718190 115753938 3% /mnt/lustre03[OST:27] lustre03-OST001c_UUID 119472128 3718873 115753255 3% /mnt/lustre03[OST:28] lustre03-OST001d_UUID 119472128 3646678 115825450 3% /mnt/lustre03[OST:29] filesystem summary: 507510784 130371598 377139186 26% /mnt/lustre03 |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
|
Frederik, 1) The output of "lctl get_param -n obdfilter.*.lfsck_layout" on all the OSS nodes looks normal. Thanks! |
| Comment by Frederik Ferner (Inactive) [ 18/Aug/15 ] |
For a random OST, I've looked random other OSTs and a few of the oi.16.xx files and all I checked had the same size of 8192, let me know if you want to see the output for all. [bnh65367@cs04r-sc-oss03-04 ~]$ for i in $(seq 1 63) ; do sudo debugfs -c -R "stat oi.16.$i" /dev/mapper/ost_lustre03_23 2>/dev/null ; done Inode: 18 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218603 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 12 00 00 00 00 00 00 00 6b 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x12:0x489c8a6b:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341122-6341123 Inode: 19 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218604 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 13 00 00 00 00 00 00 00 6c 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x13:0x489c8a6c:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341124-6341125 Inode: 20 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218605 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 14 00 00 00 00 00 00 00 6d 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x14:0x489c8a6d:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341126-6341127 Inode: 21 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218606 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 15 00 00 00 00 00 00 00 6e 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x15:0x489c8a6e:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341128-6341129 Inode: 22 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218607 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 16 00 00 00 00 00 00 00 6f 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x16:0x489c8a6f:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341130-6341131 Inode: 23 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218608 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 17 00 00 00 00 00 00 00 70 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x17:0x489c8a70:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341132-6341133 Inode: 24 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218609 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 18 00 00 00 00 00 00 00 71 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x18:0x489c8a71:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341134-6341135 Inode: 25 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218610 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 19 00 00 00 00 00 00 00 72 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x19:0x489c8a72:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341136-6341137 Inode: 26 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218611 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1a 00 00 00 00 00 00 00 73 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1a:0x489c8a73:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341138-6341139 Inode: 27 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218612 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1b 00 00 00 00 00 00 00 74 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1b:0x489c8a74:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341140-6341141 Inode: 28 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218613 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1c 00 00 00 00 00 00 00 75 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1c:0x489c8a75:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341142-6341143 Inode: 29 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218614 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1d 00 00 00 00 00 00 00 76 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1d:0x489c8a76:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341144-6341145 Inode: 30 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218615 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1e 00 00 00 00 00 00 00 77 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1e:0x489c8a77:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341146-6341147 Inode: 31 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218616 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 1f 00 00 00 00 00 00 00 78 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x1f:0x489c8a78:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341148-6341149 Inode: 32 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218617 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 20 00 00 00 00 00 00 00 79 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x20:0x489c8a79:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341150-6341151 Inode: 33 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218618 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 21 00 00 00 00 00 00 00 7a 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x21:0x489c8a7a:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341152-6341153 Inode: 34 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218619 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 22 00 00 00 00 00 00 00 7b 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x22:0x489c8a7b:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341154-6341155 Inode: 35 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218620 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:37c32c50 -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 23 00 00 00 00 00 00 00 7c 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x23:0x489c8a7c:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341156-6341157 Inode: 36 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218621 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 24 00 00 00 00 00 00 00 7d 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x24:0x489c8a7d:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341158-6341159 Inode: 37 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218622 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 25 00 00 00 00 00 00 00 7e 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x25:0x489c8a7e:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341160-6341161 Inode: 38 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218623 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 26 00 00 00 00 00 00 00 7f 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x26:0x489c8a7f:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341162-6341163 Inode: 39 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218624 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 27 00 00 00 00 00 00 00 80 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x27:0x489c8a80:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341164-6341165 Inode: 40 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218625 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 28 00 00 00 00 00 00 00 81 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x28:0x489c8a81:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341166-6341167 Inode: 41 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218626 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 29 00 00 00 00 00 00 00 82 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x29:0x489c8a82:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341168-6341169 Inode: 42 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218627 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2a 00 00 00 00 00 00 00 83 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2a:0x489c8a83:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341170-6341171 Inode: 43 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218628 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2b 00 00 00 00 00 00 00 84 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2b:0x489c8a84:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341172-6341173 Inode: 44 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218629 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2c 00 00 00 00 00 00 00 85 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2c:0x489c8a85:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341174-6341175 Inode: 45 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218630 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2d 00 00 00 00 00 00 00 86 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2d:0x489c8a86:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341176-6341177 Inode: 46 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218631 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2e 00 00 00 00 00 00 00 87 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2e:0x489c8a87:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341178-6341179 Inode: 47 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218632 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 2f 00 00 00 00 00 00 00 88 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x2f:0x489c8a88:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341180-6341181 Inode: 48 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218633 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00 89 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x30:0x489c8a89:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341182-6341183 Inode: 49 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218634 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 8a 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x31:0x489c8a8a:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341184-6341185 Inode: 50 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218635 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 32 00 00 00 00 00 00 00 8b 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x32:0x489c8a8b:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341186-6341187 Inode: 51 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218636 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 8c 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x33:0x489c8a8c:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341188-6341189 Inode: 52 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218637 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 34 00 00 00 00 00 00 00 8d 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x34:0x489c8a8d:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341190-6341191 Inode: 53 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218638 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 35 00 00 00 00 00 00 00 8e 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x35:0x489c8a8e:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341192-6341193 Inode: 54 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218639 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 36 00 00 00 00 00 00 00 8f 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x36:0x489c8a8f:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341194-6341195 Inode: 55 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218640 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 37 00 00 00 00 00 00 00 90 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x37:0x489c8a90:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341196-6341197 Inode: 56 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218641 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 38 00 00 00 00 00 00 00 91 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x38:0x489c8a91:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341198-6341199 Inode: 57 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218642 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 39 00 00 00 00 00 00 00 92 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x39:0x489c8a92:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341200-6341201 Inode: 58 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218643 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3a 00 00 00 00 00 00 00 93 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3a:0x489c8a93:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341202-6341203 Inode: 59 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218644 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3b 00 00 00 00 00 00 00 94 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3b:0x489c8a94:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341204-6341205 Inode: 60 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218645 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3c 00 00 00 00 00 00 00 95 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3c:0x489c8a95:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341206-6341207 Inode: 61 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218646 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3d 00 00 00 00 00 00 00 96 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3d:0x489c8a96:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341208-6341209 Inode: 62 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218647 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3e 00 00 00 00 00 00 00 97 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3e:0x489c8a97:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341210-6341211 Inode: 63 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218648 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 3f 00 00 00 00 00 00 00 98 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x3f:0x489c8a98:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341212-6341213 Inode: 64 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218649 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 99 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x40:0x489c8a99:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341214-6341215 Inode: 65 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218650 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 41 00 00 00 00 00 00 00 9a 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x41:0x489c8a9a:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341216-6341217 Inode: 66 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218651 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 42 00 00 00 00 00 00 00 9b 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x42:0x489c8a9b:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341218-6341219 Inode: 67 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218652 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 9c 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x43:0x489c8a9c:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341220-6341221 Inode: 68 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218653 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 44 00 00 00 00 00 00 00 9d 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x44:0x489c8a9d:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341222-6341223 Inode: 69 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218654 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 45 00 00 00 00 00 00 00 9e 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x45:0x489c8a9e:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341224-6341225 Inode: 70 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218655 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 46 00 00 00 00 00 00 00 9f 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x46:0x489c8a9f:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341226-6341227 Inode: 71 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218656 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 47 00 00 00 00 00 00 00 a0 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x47:0x489c8aa0:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341228-6341229 Inode: 72 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218657 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 a1 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x48:0x489c8aa1:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341230-6341231 Inode: 73 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218658 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 49 00 00 00 00 00 00 00 a2 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x49:0x489c8aa2:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341232-6341233 Inode: 74 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218659 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4a 00 00 00 00 00 00 00 a3 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4a:0x489c8aa3:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341234-6341235 Inode: 75 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218660 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4b 00 00 00 00 00 00 00 a4 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4b:0x489c8aa4:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341236-6341237 Inode: 76 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218661 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4c 00 00 00 00 00 00 00 a5 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4c:0x489c8aa5:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341238-6341239 Inode: 77 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218662 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4d 00 00 00 00 00 00 00 a6 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4d:0x489c8aa6:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341240-6341241 Inode: 78 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218663 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 a7 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4e:0x489c8aa7:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341242-6341243 Inode: 79 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218664 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 4f 00 00 00 00 00 00 00 a8 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x4f:0x489c8aa8:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341244-6341245 Inode: 80 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1218218665 Version: 0x00000000:00000000 User: 0 Group: 0 Size: 8192 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 16 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 atime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 mtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 crtime: 0x53f37936:380035fc -- Tue Aug 19 17:20:06 2014 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "04 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 a9 8a 9c 48 00 00 00 00 " (24) lma: fid=[0x50:0x489c8aa9:0x0] compat=4 incompat=0 EXTENTS: (0-1):6341246-6341247 [bnh65367@cs04r-sc-oss03-04 ~]$
We're running with 2.7.0 + a few patches, git --oneline extract below: 03ce777 |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
|
Let's analysis something based on the known information: Generally, if there are some orphan OST-objects, then the former case should be larger than the latter case. But the real case does not support that. So the possibility of orphan OST-objects caused the "du/df" difference should be very little, unless there are quite huge orphan OST-objects. Andreas, how do you think? |
| Comment by Frederik Ferner (Inactive) [ 18/Aug/15 ] |
|
(looks like we've been thinking along similar lines...) I've also done one other checks that I could think of: Our file system is configured with a default strip count of '1', adding up the numbers of used inodes on the OSTs (lfs df -i) and comparing this with the number of files reported by rbh-du turns out to match fairly closely (within 1%). |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
I do NOT want to say the "rbh-du" is wrong. But I wonder why it can be much faster than the normal "du". I am not familiar with the "rbh-du". If it does not scan the whole system, how to guarantee it can return the latest system usage? As for you said "In the past I did verify that rbh-du and standard du basically agree", I am not sure on how large system you made such verification. But is it possible that there may be some obvious difference on very large system? |
| Comment by Frederik Ferner (Inactive) [ 18/Aug/15 ] |
|
rbh-du is the robinhood tool (https://github.com/cea-hpc/robinhood/wiki), after the initial scan it uses Lustre changelogs to keep the information current in a database, that's why is much much faster. The "verificiation" was done on the same file system we are currently investigating by running du and comparing the result with the robinhood output at the time when du completed, they both agreed. |
| Comment by Gerrit Updater [ 18/Aug/15 ] |
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/16020 |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
The memory pressure on the MDS may be caused by the (too much) LFSCK async RPCs. This patch will avoid unexpected LFSCK preparing, as then reduce the memory pressure on the MDS. Please apply the patch on the MDS, and re-run the layout LFSCK as following: lctl lfsck_start -M lustre03-MDT0000 -t layout -r -o -w 512 Please note "-w" option, that will help the LFSCK flow control. |
| Comment by nasf (Inactive) [ 18/Aug/15 ] |
The idea of using changelog to accelerate "du" is good. If the system always runs in health status, the "rbh-du" and normal "du" should agree with each other. But if there are something wrong during file close for write/punch, or recording the changelog itself also maybe failed, then Lustre changelog may miss some write/punch operations, and cause "rbh-du" to report staled space usage. So I suggest to run the normal "du" for double check, that will scan the whole system, and obtain the space usage from the OSTs. I think that you have already started the normal "du", let's wait for the result. If you have not start the normal "du" yet, then to accelerate normal "du" speed, you can run "du" on multiple clients in parallel. For each client, only "du" some sub-directory, then summary all the involved clients' "du" results. |
| Comment by Frederik Ferner (Inactive) [ 18/Aug/15 ] |
|
I will look at applying the patch tomorrow. You were right, I had started normal du (du -sxk running as root) and both instances have completed now and agree within a few kB: 261351943900 /mnt/lustre03 Which seems to match the rbh-du output from yesterday reasonably well, considering that we are currently trying to remove as much data as we can (within our policy). The most recent rbh-du status is (in kB): file count:109780676, size:244348429416, spc_used:233872570099 |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
So how is the latest du/df difference after the normal "du" completed? |
| Comment by Frederik Ferner (Inactive) [ 19/Aug/15 ] |
|
Ah, sorry providing df output would have been the obvious addition. du/df is pretty much unchanged, latest df: [bnh65367@cs04r-sc-serv-92 ~]$ df /mnt/lustre03 |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
So there are still ~120TB space difference. Consider your OSTs' space usage (based on the "lfs df" ouput), it approximately means that every OST lost about 4TB space. When your system is offline for maintaining, please mount one OST as ldiskfs, and run normal "du" on such OST directly. For example: Because the OST only contains about 1/30 objects of the whole system, it should not take a long time for the normal "du" on the OST. |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
On the other hand, you can choose more OSTs on different OSS nodes to make above check in parallel. Then we can know whether different OSTs will have similar space overhead. If some OST's space overhead is some large, such as hundreds of GB, then please check every OST. |
| Comment by Frederik Ferner (Inactive) [ 19/Aug/15 ] |
|
The file system is currently down and I did run du -xk --max-depth=2 on all ldiskfs mounted OSTs. The result is fairly consistent across all OSTs. I can attach the full logs if you want. Doing the sums on one OST as an example: 92 /lustre/lustre03/ost_0/CONFIGS 16 /lustre/lustre03/ost_0/lost+found 136 /lustre/lustre03/ost_0/O/1 8 /lustre/lustre03/ost_0/O/2 8411888660 /lustre/lustre03/ost_0/O/0 136 /lustre/lustre03/ost_0/O/10 132 /lustre/lustre03/ost_0/O/200000008 136 /lustre/lustre03/ost_0/O/200000003 8411889212 /lustre/lustre03/ost_0/O 4 /lustre/lustre03/ost_0/REMOTE_PARENT_DIR 4 /lustre/lustre03/ost_0/LFSCK 36 /lustre/lustre03/ost_0/quota_slave 8411890432 /lustre/lustre03/ost_0 S1-S2: 8411890432-8411889212=1220 However even for the ldiskfs mounted OST, df reports 12825556952 of 1K blocks used on the same file system, so already on that level we have the 4TB that are lost/invisible. For reference, here is the tune2fs -l output for this device, I don't see any issues there, but maybe I'm missing something? sudo tune2fs -l /dev/mapper/ost_lustre03_0 tune2fs 1.42.12.wc1 (15-Sep-2014) Filesystem volume name: lustre03-OST0000 Last mounted on: / Filesystem UUID: df65e0d0-e76c-46d2-827b-9a0f9035f3a4 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent mmp sparse_super large_file uninit_bg Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 119472128 Block count: 3823108096 Reserved block count: 38231080 Free blocks: 608894194 Free inodes: 115787562 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 112 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 1024 Inode blocks per group: 64 Filesystem created: Wed Jun 22 15:51:37 2011 Last mount time: Wed Aug 19 10:09:29 2015 Last write time: Wed Aug 19 11:31:53 2015 Mount count: 62 Maximum mount count: 21 Last checked: Wed Jun 22 15:51:37 2011 Check interval: 15552000 (6 months) Next check after: Mon Dec 19 14:51:37 2011 Lifetime writes: 33 TB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: a0f8eb19-4c4a-47ea-b4f2-ee5defc962e1 Journal backup: inode blocks MMP block number: 1097 MMP update interval: 5 [bnh65367@cs04r-sc-oss03-01 ~]$ |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
According to the local "du" output, we can say that it is NOT the Lustre system files/logs that caused the space leak. Because the system backend files/logs only occupy about 1220KB, that can be ignored compared with 4TB. On the other hand, the local "df" shows that 12825556952KB space used, but local "du" only reports 8411890432KB. Means that it is NOT the orphan OST-objects that caused the space leak. But it maybe related with orphan ldiskfs-inodes/blocks. Please run e2fsck on the OST device for verification. To be safe, please use "dryrun" mode. |
| Comment by Frederik Ferner (Inactive) [ 19/Aug/15 ] |
|
Both OSTs I've just check (read-only) are reporting many block bitmap differences in pass 5 (full output for each of them is rather large, ~60MB uncompressed, 18MB compressed each, so I'm not attaching them, let me know if you want them...) e2fsck 1.42.12.wc1 (15-Sep-2014) Based on that, would you recommend to run "e2fsck -p" for all OSTs? Or on only those two initially? Any other command? (And yes, I believe we've updated e2fsck when we upgraded to 2.7 and I've just been able to get to downloads.hpdd.intel.com long enough to verify that 1.42.12.wc1 seems to be the latest...) Frederik |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
I would say "yes" for "e2fsck -p" on one OST firstly to check whether the space leak can be recovered by the e2fsck. But to be safe, it is better to backup the OST before the repairing. On the other hand, I have one question: if it is true that every OST has ~4TB space leak because of "Block bitmap differences", then how this happened? If it is caused by some random failures, then why not some OST lost 7TB and some OST lost 1TB? So would please to collect the info about how much space leaked on every OST? If all of them have the similar space leak, then I prefer to understand the reason firstly to avoid improper repairing. |
| Comment by Frederik Ferner (Inactive) [ 19/Aug/15 ] |
|
It certainly looks like every OST has the same sort of difference for du/df, so doesn't look random: du (I should probably say none of them had much data on the top level) [bnh65367@ws104 bnh65367]$ grep -E ".*/lustre/lustre03/.*/O[^/]*$" oss03-*-checks.txt oss03-01-checks.txt:cs04r-sc-oss03-01: 8411889212 /lustre/lustre03/ost_0/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8476774800 /lustre/lustre03/ost_1/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8357603988 /lustre/lustre03/ost_2/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8350516728 /lustre/lustre03/ost_3/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8398291652 /lustre/lustre03/ost_4/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8408603604 /lustre/lustre03/ost_5/O oss03-01-checks.txt:cs04r-sc-oss03-01: 8500699476 /lustre/lustre03/ost_6/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8442811392 /lustre/lustre03/ost_10/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8427939676 /lustre/lustre03/ost_11/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8166657184 /lustre/lustre03/ost_12/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8317848496 /lustre/lustre03/ost_13/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8361699964 /lustre/lustre03/ost_14/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8406544328 /lustre/lustre03/ost_7/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8374354340 /lustre/lustre03/ost_8/O oss03-02-checks.txt:cs04r-sc-oss03-02: 8396160196 /lustre/lustre03/ost_9/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8387479956 /lustre/lustre03/ost_15/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8347133816 /lustre/lustre03/ost_16/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8315323460 /lustre/lustre03/ost_17/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8490110544 /lustre/lustre03/ost_18/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8421718608 /lustre/lustre03/ost_19/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8593474944 /lustre/lustre03/ost_20/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8368913028 /lustre/lustre03/ost_21/O oss03-03-checks.txt:cs04r-sc-oss03-03: 8487284272 /lustre/lustre03/ost_22/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8349964628 /lustre/lustre03/ost_23/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8385949148 /lustre/lustre03/ost_24/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8475699168 /lustre/lustre03/ost_25/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8317572224 /lustre/lustre03/ost_26/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8319496876 /lustre/lustre03/ost_27/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8445175912 /lustre/lustre03/ost_28/O oss03-04-checks.txt:cs04r-sc-oss03-04: 8387123652 /lustre/lustre03/ost_29/O df: [bnh65367@ws104 bnh65367]$ grep "% /lustre/lustre03/o" oss03-0*-checks.txt oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12825556952 2282652456 85% /lustre/lustre03/ost_0 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12752407720 2355801688 85% /lustre/lustre03/ost_1 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12872497012 2235712396 86% /lustre/lustre03/ost_2 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12879766228 2228443180 86% /lustre/lustre03/ost_3 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12775778084 2332431324 85% /lustre/lustre03/ost_4 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12881962520 2226246888 86% /lustre/lustre03/ost_5 oss03-01-checks.txt:cs04r-sc-oss03-01: 15261133728 12717896644 2390312764 85% /lustre/lustre03/ost_6 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12968537252 2139672156 86% /lustre/lustre03/ost_10 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12998294692 2109914716 87% /lustre/lustre03/ost_11 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12959228716 2148980692 86% /lustre/lustre03/ost_12 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 13058966988 2049242420 87% /lustre/lustre03/ost_13 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12851358948 2256850460 86% /lustre/lustre03/ost_14 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12882729284 2225480124 86% /lustre/lustre03/ost_7 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12808224420 2299984988 85% /lustre/lustre03/ost_8 oss03-02-checks.txt:cs04r-sc-oss03-02: 15261133728 12899361172 2208848236 86% /lustre/lustre03/ost_9 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12843122972 2265086436 86% /lustre/lustre03/ost_15 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12876906128 2231303280 86% /lustre/lustre03/ost_16 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12979969156 2128240252 86% /lustre/lustre03/ost_17 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12723623176 2384586232 85% /lustre/lustre03/ost_18 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12839696676 2268512732 85% /lustre/lustre03/ost_19 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12972879920 2135329488 86% /lustre/lustre03/ost_20 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12778815012 2329394396 85% /lustre/lustre03/ost_21 oss03-03-checks.txt:cs04r-sc-oss03-03: 15261133728 12880579688 2227629720 86% /lustre/lustre03/ost_22 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12872707416 2235501992 86% /lustre/lustre03/ost_23 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12904916800 2203292608 86% /lustre/lustre03/ost_24 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12956118728 2152090680 86% /lustre/lustre03/ost_25 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12877741672 2230467736 86% /lustre/lustre03/ost_26 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12836857632 2271351776 85% /lustre/lustre03/ost_27 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 12864834860 2243374548 86% /lustre/lustre03/ost_28 oss03-04-checks.txt:cs04r-sc-oss03-04: 15261133728 13055100536 2053108872 87% /lustre/lustre03/ost_29 When you suggested backup: are you thinking of block level backup (dd) or file level (tar or something similar)? We're rather keen to free up the space but obviously don't want to loose data... |
| Comment by nasf (Inactive) [ 19/Aug/15 ] |
|
I would suggest to run “e2fsck -n” on another OST. If report the same failure position, then we have to suspect that the issue has been there since format time. As for the backup, because your system has more than 2/3 used, so device level backup (such as “dd”) will be faster than file-level backup (such as “tar”). |
| Comment by Frederik Ferner (Inactive) [ 20/Aug/15 ] |
|
after the jira outage, here is a summary what we've done in the mean time. Please add if I missed anything of importance. "e2fsck -n" on all other OSTs where we tried it reported the same approximate amount of block bitmap difference, though with different locations. We decided to create a block level (dd) backup of one OST, ran interactive "e2fsck" on this, fixing the block bitmap differences but interrupting this when following this we had a very large number of "free block count wrong" instances. Running another read-only e2fsck showed ~100000 of these. After this we ran "e2fsck -p" on that OST, which seems to free up a large amount of space (reducing the used space to 55% when mounting as ldiskfs. There was nothing in lost+found after mounting as ldiskfs. Bringing the MDT and just this OST back, we were able to access files on the OST and confirm the were as expected. We are currently running "e2fsck -p ;e2fsck -n -f" on all OSTs (one OST per OSS at a time). |
| Comment by Dave Bond (Inactive) [ 21/Aug/15 ] |
|
Hello This morning after mounting as ldiskfs and looking at the disk fill. It looks like this has been successful over the OST's. All now showing 56 / 57 % The total now shows the following as reported by the client cs04r-sc-mds03-01-10ge@tcp:cs04r-sc-mds03-02-10ge@tcp:/lustre03
427T 235T 188T 56% /mnt/lustre03
A small concern when mounting the OST's there were a few occurrences of an LBUG when recovery was happening kernel:LustreError: 24580:0:(ldlm_lib.c:2277:target_queue_recovery_request()) ASSERTION( req->rq_export->exp_lock_replay_needed ) failed: kernel:LustreError: 24580:0:(ldlm_lib.c:2277:target_queue_recovery_request()) LBUG After a few attempts things seem to have settled down and the file system is mounted. But I would worry that this would happen in the future, can you please advise? |
| Comment by Oleg Drokin [ 21/Aug/15 ] |
|
It seems this is I see you have 2.5.3 clients that have no patch from that ticket and since the patch was client-side, servers are still exposed (similar to |
| Comment by nasf (Inactive) [ 30/Aug/15 ] |
|
Any feedback about this ticket? any left issue to be resolve or can we close it? Thanks! |
| Comment by Peter Jones [ 30/Aug/15 ] |
|
Fan Yong Dave had asked above as to how they could avoid this situation reoccurring in the future. Is there any advice you can give there? Peter |
| Comment by nasf (Inactive) [ 30/Aug/15 ] |
|
Oleg has already answered their question about "target_queue_recovery_request()) LBUG". As for how to avoid the space leak, honestly, we do not know the root reason for their former space leak. There are some possible reasons may cause that, for example: the in-processing unlink/destroy is interrupted; the storage outage unexpectedly; and so on. |
| Comment by Frederik Ferner (Inactive) [ 10/Sep/15 ] |
|
Sorry for the delay, we've been busy. I agree that the LBUG is understood, I had thought we had only 2.7 clients in the file system but it turned out that we had missed a small number. They have now all been upgraded to 2.7. I'm not sure I can believe in interrupted unlink/destroy or unexpected storage outage being the cause of the space leak at that scale as the file system in general seems fairly stable and we don't have that many instabilities. However if there is nothing else we can do to understand the root cause, I guess there is nothing really gained from leaving this ticket open. We will continue to monitor our file systems and will open a new ticket if this appears to be happening again (hopefully with more time to debug before we run out of space than was the case in this instance.) Thanks everyone for their help. Frederik |
| Comment by nasf (Inactive) [ 28/Sep/15 ] |
|
The original issues have been resolved via e2fsck. |
)