[LU-8218] lfsck not able to recover files lost from MDT

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Nathan Dauchy (Inactive)

Description

My understanding is that lfsck in lustre-2.7 should be able to handle lost file information on the MDT, as long as the objects are still on the OSTs. However, a simple test to simulate this is not recovering the files. Shouldn't it at least be able to put them into lost+found? Or am I misunderstanding the capabilities of lfsck? Or is the following test case invalid in some way?

On the client, just create some test files...

# cd /mnt/lustre/client/lfscktest
# echo foo > foo
# mkdir bar
# echo baz > bar/baz

# lfs getstripe foo bar/baz
foo
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_pattern:        1
lmm_layout_gen:     0
lmm_stripe_offset:  9
    obdidx         objid         objid         group
         9            460962          0x708a2                 0

bar/baz
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_pattern:        1
lmm_layout_gen:     0
lmm_stripe_offset:  12
    obdidx         objid         objid         group
        12            460866          0x70842                 0

# sync

On the MDS, simulate the MDT losing the information, such as could happen through restoring from a slightly outdated MDT backup...

# umount /mnt/lustre/nbptest-mdt
# mount -t ldiskfs /dev/mapper/nbptest--vg-mdttest /mnt/lustre/nbptest-mdt
# cd /mnt/lustre/nbptest-mdt/ROOT

# ls -ld lfscktest lfscktest/*
drwxr-xr-x+ 3 root root 4096 May 30 08:15 lfscktest
drwxr-xr-x+ 2 root root 4096 May 30 08:15 lfscktest/bar
-rw-r--r--  1 root root    0 May 30 08:14 lfscktest/foo

# rm -rf lfscktest/*

# cd
# umount /mnt/lustre/nbptest-mdt
# mount -t lustre /dev/mapper/nbptest--vg-mdttest /mnt/lustre/nbptest-mdt

Now check the filesystem...

# lctl clear
# lctl debug_daemon start /var/log/lfsck.debug
# lctl lfsck_start -A -M nbptest-MDT0000 -c on -C on -o
Started LFSCK on the device nbptest-MDT0000: scrub layout namespace

# lctl get_param -n osd-ldiskfs.*.oi_scrub | grep status
status: init
status: completed

# lctl debug_daemon stop
# lctl debug_file /var/log/lfsck.debug | egrep -v " (NRS|RPC) " > /var/log/lfsck.log

And look back on the client...

# cd /mnt/lustre/client/         

# ls -la lfscktest/
total 8
drwxr-xr-x+ 2 root root 4096 May 30 08:22 .
drwxr-xr-x+ 9 root root 4096 May 30 08:14 ..

# ls -la .lustre/lost+found/MDT0000
total 8
drwx------+ 3 root root 4096 May 27 10:44 .
dr-x------+ 3 root root 4096 May 27 09:01 ..

Notice that there is no sign of the files being restored anywhere. Nor do I find any mention of the object ID's in the lfsck.log file.

Note that running lfsck_start with the "-t layout" option did not change the behaviour either.

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20659/
Subject: LU-8218 osd: handle stale OI mapping for non-restore case
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cecde8bdb4913fd4405d425b0bf3aead03181e9d

[nasf (Inactive)]

The patch 20659 should be landed to master, and then be ported to the other branches.

[Jay Lan (Inactive)]

Hi Fan Yong,

Do you intend to land http://review.whamcloud.com/20659 to master and future releases or to provide a workaround for us?

[nasf (Inactive)]

That is what we should expect, even with the patch, right? There is no way to determine the object's path once it is lost from the ROOT tree on the MDT?

Yes, that is what we can do now. The path information is stored as linkEA ("trusted.link") in the MDT-object. There is no other backup in the system. So if the MDT-object itself lost, then the LFSCK cannot know its original location, and have to put it under .luster/lost+found/

[Nathan Dauchy (Inactive)]

Fan Yong, thank you for the patch! I haven't had a chance to test with a new build yet, but did do a quick check of running lfsck after "rm -f oi.16.*" under ldiskfs. The lfsck then resulted in files like the following in ".lustre/lost+found/MDT0000/":

.lustre/lost+found/MDT0000/[0x200003ab0:0x1:0x0]-R-0

That is what we should expect, even with the patch, right? There is no way to determine the object's path once it is lost from the ROOT tree on the MDT?

[nasf (Inactive)]

Nathan,

Above patch may be not perfect solution, but it should be enough to resolve your case.

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20659
Subject: LU-8218 osd: handle stale OI mapping for non-restore case
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 31cf77414ad4f88c28d6eb2be54b32a7ec399ab7

[nasf (Inactive)]

The workaround for your special case is that if you want to remove some MDT-object under "ldiskfs" mode directly, then please remove the OI files also.

[Nathan Dauchy (Inactive)]

Just to clarify the status of this ticket... we are on hold waiting for a new phase of scanning to be added to lfsck?

In the meantime, is there a workaround we can use as part of the MDT recovery procedure when getting such stale mappings is expected? Can we mount as ldiskfs and manually check or clean things up?

[nasf (Inactive)]

From the LFSCK view, the case of removing MDT-object directly without destroy the OI mapping is indistinguishable from the case of MDT file-level backup/restore. When the OSD tries to locate the local object/inode via the ino# that is obtained from the stale OI mapping, it does not know whether the real MDT-object exists or not. The possible solution is that the OI scrub should make double scanning: the first phase scanning is inode table based to scan all know object on the device; the second phase scanning is OI files based to find out all staled OI mappings. Currently, it only does the first phase scanning.

[Nathan Dauchy (Inactive)]

OK... I can test that, but what if this was a "real" case of MDT corruption where only the files were lost? Is a new feature or phase in lfsck needed to manage the stale OI mappings?