[LU-11186] Cannot create files on MDT1, ofd_precreate_objects()) snx11205-OST0001:0xffffffdd:11811161088 hit the OBIF_MAX_OID (1<<32)! Created: 27/Jul/18  Updated: 29/Jan/22  Resolved: 09/Aug/18

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.11.0
Fix Version/s: Lustre 2.12.0, Lustre 2.10.7

Type: Bug Priority: Critical
Reporter: Alexander Boyko Assignee: Alexander Boyko
Resolution: Fixed Votes: 0
Labels: patch

Issue Links:
Duplicate
is duplicated by LU-6417 Test cases for sequence roll over. Resolved
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description    
Jul  7 03:27:16 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xffffffdd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:16 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:18 snx11205n005 kernel: LustreError: 8958:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:18 snx11205n005 kernel: LustreError: 8958:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:662:osp_precreate_send()) snx11205-OST0001-osc-MDT0001: precreate fid [0x2c0000400:0xfffffffc:0x0] < local used fid [0x2c0000400:0xfffffffc:0x0]: rc = -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:1287:osp_precreate_thread()) snx11205-OST0001-osc-MDT0001: cannot precreate objects: rc = -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 13467:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 9294:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:22 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:22 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:662:osp_precreate_send()) snx11205-OST0001-osc-MDT0001: precreate fid [0x2c0000400:0xfffffffc:0x0] < local used fid [0x2c0000400:0xfffffffc:0x0]: rc = -116
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:1287:osp_precreate_thread()) snx11205-OST0001-osc-MDT0001: cannot precreate objects: rc = -116
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 9305:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:32 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:32 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28Jul  7 03:27:16 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xffffffdd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:16 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:18 snx11205n005 kernel: LustreError: 8958:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:18 snx11205n005 kernel: LustreError: 8958:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:662:osp_precreate_send()) snx11205-OST0001-osc-MDT0001: precreate fid [0x2c0000400:0xfffffffc:0x0] < local used fid [0x2c0000400:0xfffffffc:0x0]: rc = -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:1287:osp_precreate_thread()) snx11205-OST0001-osc-MDT0001: cannot precreate objects: rc = -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 13467:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:18 snx11205n003 kernel: LustreError: 9294:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:22 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:22 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:662:osp_precreate_send()) snx11205-OST0001-osc-MDT0001: precreate fid [0x2c0000400:0xfffffffc:0x0] < local used fid [0x2c0000400:0xfffffffc:0x0]: rc = -116
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 24598:0:(osp_precreate.c:1287:osp_precreate_thread()) snx11205-OST0001-osc-MDT0001: cannot precreate objects: rc = -116
Jul  7 03:27:22 snx11205n003 kernel: LustreError: 9305:0:(osp_precreate.c:1339:osp_precreate_ready_condition()) snx11205-OST0001-osc-MDT0001: precreate failed opd_pre_status -116
Jul  7 03:27:32 snx11205n005 kernel: LustreError: 9052:0:(ofd_objects.c:217:ofd_precreate_objects()) snx11205-OST0001:0xfffffffd:11811161088 hit the OBIF_MAX_OID (1<<32)!
Jul  7 03:27:32 snx11205n005 kernel: LustreError: 9052:0:(ofd_dev.c:1764:ofd_create_hdl()) snx11205-OST0001: unable to precreate: rc = -28

the reproducer
I`m changing manual last id to the end of range for ost1 and mdt2 for a less file creations.

 MDSCOUNT=2 sh lustre/tests/llmount.sh
  lfs mkdir -i1 -c1 /mnt/lustre/mdt2
 lfs setstripe -i0 -c1 /mnt/lustre/mdt2
  lustre/tests/createmany -o /mnt/lustre/mdt2/file_ 100
 MDSCOUNT=2 sh lustre/tests/llmountcleanup.sh
mount -o loop -t ldiskfs /tmp/lustre-ost1 /mnt/lustre
hexedit /mnt/lustre/O/280000400/LAST_ID   /change zeroes to ff, so it looks like 83 ff ff ff 
 umount /mnt/lustre
mount -o loop -t ldiskfs /tmp/lustre-mdt2 /mnt/lustre
 hexedit /mnt/lustre/lov_objid    //change zeroes to ff at first u32, looks like 63 ff ff ff
 umount /mnt/lustre
 NOFORMAT=1 MDSCOUNT=2 sh lustre/tests/llmount.sh
 lustre/tests/createmany -o /mnt/lustre/mdt2/file1_ 100
 lustre/tests/createmany -o /mnt/lustre/mdt2/file2_ 100


 Comments   
Comment by Gerrit Updater [ 27/Jul/18 ]

Alexandr Boyko (c17825@cray.com) uploaded a new patch: https://review.whamcloud.com/32891
Subject: LU-11186 ofd: fix for a final oid at sequence
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2fb8f999d52da4e43ad37ee1496ab01494026b76

Comment by Gerrit Updater [ 09/Aug/18 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32891/
Subject: LU-11186 ofd: fix for a final oid at sequence
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b724079edc5b66e1046b5760a6bad3045e9a9260

Comment by Peter Jones [ 09/Aug/18 ]

Landed for 2.12

Comment by Gerrit Updater [ 04/Sep/18 ]

Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33111
Subject: LU-11186 ofd: fix for a final oid at sequence
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 427f025b821def683d751b83fa022e17214e2c58

Comment by Gerrit Updater [ 19/Jan/19 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33111/
Subject: LU-11186 ofd: fix for a final oid at sequence
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 1e855d9762c2f195010f46ff14bcdef0c4c733f4

Generated at Sat Feb 10 02:41:41 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.