[LU-14119] FID-in-LMA [fid1] does not match the object self-fid [fid2]

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.12.7, Lustre 2.15.0
• Labels: llnl
• Environment: zfs-0.7 ; 3.10.0-1127.18.2.1chaos.ch6.x86_64 ; lustre-2.12.5_5.llnl-1.ch6.x86_64
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Olaf Faaland

Description

In console log on the MDS:

Nov  5 09:16:10 zinc1 kernel: LustreError: 16479:0:(osd_object.c:481:osd_check_lma()) lsh-MDT0000: FID-in-LMA [0x200044dd2:0xcbc2:0x0] does not match the object self-fid [0x20003a324:0x8267:0x0]

One FID resolves with fid2path, and a stat of the path returned is successful:

[root@oslic8:~]# lfs fid2path /p/lustre2/ 0x200044dd2:0xcbc2:0x0
/p/lustre2/settgast/GEOSX/src/cmake/blt/tests/internal/src/combine_static_library_test/Foo3.cpp
[root@oslic8:~]# stat /p/lustre2/settgast/GEOSX/src/cmake/blt/tests/internal/src/combine_static_library_test/Foo3.cpp
  File: '/p/lustre2/settgast/GEOSX/src/cmake/blt/tests/internal/src/combine_static_library_test/Foo3.cpp'
  Size: 445       	Blocks: 56         IO Block: 4194304 regular file
Device: a2f6a642h/2734073410d	Inode: 144119920358116290  Links: 1
Access: (0600/-rw-------)  Uid: (56443/settgast)   Gid: (56443/settgast)
Access: 2020-03-29 21:07:15.000000000 -0700
Modify: 2020-03-29 21:07:16.000000000 -0700
Change: 2020-03-29 21:07:16.000000000 -0700
 Birth: -

But the other fid2path hangs:

[root@oslic8:~]# lfs fid2path /p/lustre2/ 0x20003a324:0x8267:0x0 &
[1] 53840

 

The hanging fid2path has this stack:

[root@oslic8:~]# cat /proc/53840/stack
[<ffffffffc17b8dd8>] ptlrpc_set_wait+0x4d8/0x800 [ptlrpc]
[<ffffffffc17b9183>] ptlrpc_queue_wait+0x83/0x230 [ptlrpc]
[<ffffffffc0e8fc99>] mdc_get_info+0x169/0x5c0 [mdc]
[<ffffffffc0e96f63>] mdc_iocontrol+0x2313/0x2c50 [mdc]
[<ffffffffc0e05b9a>] obd_iocontrol+0x7a/0x350 [lmv]
[<ffffffffc0e0dd8a>] lmv_fid2path+0x17a/0x850 [lmv]
[<ffffffffc0e0f750>] lmv_iocontrol+0x330/0x1a00 [lmv]
[<ffffffffc1138867>] ll_fid2path+0x4a7/0x7a0 [lustre]
[<ffffffffc112461b>] ll_dir_ioctl+0x7fb/0x6d80 [lustre]
[<ffffffff9fc75800>] do_vfs_ioctl+0x420/0x6d0
[<ffffffff9fc75b51>] SyS_ioctl+0xa1/0xc0
[<ffffffffa01c1112>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff 

We see this on two different file systems (at least). Both were upgraded from Lustre 2.10 to Lustre 2.12 several months ago.

Issue Links

• is related to: LU-13124 lfsck check for multiple linked file at OST
• is related to: LU-13392 FID-in-LMA does not match the object self-fid after upgrade from 2.10 to 2.12
• is related to: LU-14175 OI Scrub triggered followed by LBUG ASSERTION( idx1 == 0 || idx1 == osd->od_index ) failed
• is related to: LU-13608 MDT stuck in WAITING, abort_recov stuck too

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43265/
Subject: LU-14119 lfsck: replace dt_lookup() with dt_lookup_dir()
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 5d4356ebec16e8a08b566334d94e6422e4193e51

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43268/
Subject: LU-14119 osd: delete stale OI mapping entry
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 53db3716685b67406580048c74e21c5c33db262b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43267/
Subject: LU-14119 osd-zfs: enable LUDA_VERIFY
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 3f6c9542373dea0321883506fa9bc2c9502629e3

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43269/
Subject: LU-14119 osd: add mount option "resetoi"
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 8f1a3a63e3157642025ab674fe2ba4a72cfa3151

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43270/
Subject: LU-14119 lfsck: check linkea if it's newly added
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 3e59ed26d04da1fa2891b8f5263b0481d7c69f22

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43266/
Subject: LU-14119 mdc: set fid2path RPC interruptible
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 9055a0a1f3720aed844941213c676508bd30a69e

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43270
Subject: LU-14119 lfsck: check linkea if it's newly added
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: b54e4b3e398448a5a247f01c91acea801bf890d6

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43269
Subject: LU-14119 osd: add mount option "resetoi"
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 1c878c85172025d201c8656f9c19084469fcec91

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43268
Subject: LU-14119 osd: delete stale OI mapping entry
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 301e2c111fdbf699572c4714c462945ec0b1765b