[LU-9442] OST unable to precreate new objects after reaching 0xFFFFFFFF

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Sergey Cheremencev

Description

Issue could be simple reproduced changing lov_objid and LAST_ID to value near with 0xFFFFFFFF.
Example:

[root@devvm-2 tests]# cd /mnt/lustre
[root@devvm-2 lustre]# mkdir test1
[root@devvm-2 lustre]# lfs setstripe -i 0 -c 1 test1
[root@devvm-2 lustre]# touch test1/asdf
[root@devvm-2 lustre]# touch test1/qwerty
[root@devvm-2 lustre]# cd
[root@devvm-2 ~]# umount /mnt/lustre-mds
lustre-mds1/ lustre-mds2/ 
[root@devvm-2 ~]# umount /mnt/lustre-mds1/
[root@devvm-2 ~]# umount /mnt/lustre-ost1/
[root@devvm-2 ~]# mount -t ldiskfs -o loop /tmp/lustre-mdt1 /mnt/mds
[root@devvm-2 ~]# mount -t ldiskfs -o loop /tmp/lustre-ost1 /mnt/ost
[root@devvm-2 ~]# hexedit /mnt/mds/lov_objid 
[root@devvm-2 ~]# hexedit /mnt/ost/O/0/LAST_ID 
[root@devvm-2 ~]# hexdump -C /mnt/mds/lov_objid 
00000000  f5 ff ff ff 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000010
[root@devvm-2 ~]# hexdump -C /mnt/ost/O/0/LAST_ID 
00000000  f5 ff ff ff 00 00 00 00                           |........|
00000008
[root@devvm-2 ~]# umount /mnt/mds
[root@devvm-2 ~]# umount /mnt/ost
[root@devvm-2 ~]# mount -t lustre -o loop /tmp/lustre-mdt1 /mnt/mds
[root@devvm-2 ~]# mount -t lustre -o loop /tmp/lustre-ost1 /mnt/lustre-ost1/
[root@devvm-2 test1]# for i in `seq 1 10`; do touch file$i; done
[root@devvm-2 test1]# dmesg | tail -n 20
...
Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
Lustre: lustre-OST0000: trigger OI scrub by RPC for the [0x100000001:0x0:0x0] with flags 0x4a, rc = 0
LustreError: 4279:0:(ofd_dev.c:1775:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -115
LustreError: 4263:0:(osp_precreate.c:619:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -115
LustreError: 4263:0:(osp_precreate.c:1273:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -115
LustreError: 4351:0:(ofd_dev.c:1775:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -115

According to OST's lustre logs it fails on following:

00002000:00000040:5.0:1493820390.271480:0:4351:0:(ofd_dev.c:416:ofd_object_init()) object init, fid = [0x100000001:0x0:0x0]
00080000:00000010:5.0:1493820390.271486:0:4351:0:(osd_handler.c:338:osd_object_alloc()) kmalloced 'mo': 200 at ffff8800365270c0.
00080000:00000001:5.0:1493820390.271489:0:4351:0:(osd_handler.c:993:osd_fid_lookup()) Process entered
00000001:00000001:5.0:1493820390.271491:0:4351:0:(osd_compat.c:1285:osd_obj_spec_lookup()) Process entered
00000001:00000001:5.0:1493820390.271492:0:4351:0:(osd_compat.c:875:osd_seq_load()) Process entered
00000001:00000001:5.0:1493820390.271493:0:4351:0:(osd_compat.c:883:osd_seq_load()) Process leaving (rc=18446612135418184448 : -131938291367168 : ffff8800b902c300)
00000001:00000001:5.0:1493820390.271516:0:4351:0:(osd_compat.c:1320:osd_obj_spec_lookup()) Process leaving (rc=0 : 0 : 0)
00080000:00000001:5.0:1493820390.271517:0:4351:0:(osd_handler.c:523:osd_iget_check()) Process entered
00080000:00000001:5.0:1493820390.271518:0:4351:0:(osd_handler.c:648:osd_iget_check()) Process leaving via put (rc=0 : 0 : 0x0)
00080000:00000001:5.0:1493820390.271519:0:4351:0:(osd_handler.c:727:osd_check_lma()) Process entered
00080000:00000001:5.0:1493820390.271524:0:4351:0:(osd_handler.c:791:osd_check_lma()) Process leaving (rc=18446744073709551538 : -78 : ffffffffffffffb2)

Issue Links

• is duplicated by: LU-9631 sanity-lfsck test_18a: Expect 3 fixed on mds1, but got: 0

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/28227/
Subject: LU-9442 osp: can't create IDIF fid number > 0xFFFFFFFF
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: baa1ce2e0bbcda59d7a4acb7e67cebde402f9d95

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28227
Subject: LU-9442 osp: can't create IDIF fid number > 0xFFFFFFFF
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: eab600a7927c2e45e0dde2400d31f9bf58aed2a9

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27225/
Subject: LU-9442 osp: can't create IDIF fid number > 0xFFFFFFFF
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2dde01f1edac9e330f853c3ffee64eb43d82b7c1

[Gerrit Updater]

Sergey Cheremencev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27225
Subject: LU-9442 osp: can't create IDIF fid number > 0xFFFFFFFF
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2594290c6d550a012857b0e6a1f80a50f3b89c36

[Sergey Cheremencev]

The problem is understood. In the nearest days I will push the patch.

So don't spend time on it.