[LU-5085] ofd_create()) charlie-OST004f: unable to precreate: rc = -116

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.4.1
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Daire Byrne (Inactive)

Description

Hi,

Unfortunately we suffered a severe RAID controller failure over the weekend. We have gotten the RAID back online but even after a filesystem check we seem to be getting this on two of the OSTs. How severe is this error? I recall that previous versions of Lustre had a concept of LAST_ID which the precreated objects depended upon. Maybe this has been corrupted on these filesystems? Please advise.

May 19 15:29:15 coss5 kernel: LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. quota=on. Opts: 
May 19 15:29:16 coss5 kernel: LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. quota=on. Opts: 
May 19 15:29:16 coss5 kernel: Lustre: charlie-OST0053: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
May 19 15:29:16 coss5 kernel: Lustre: Skipped 5 previous similar messages
May 19 15:29:17 coss5 kernel: LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. quota=on. Opts: 
May 19 15:29:19 coss5 rlogind[15167]: PAM authentication failed for in.rlogind
May 19 15:29:25 coss5 kernel: Lustre: charlie-OST0053: Will be in recovery for at least 2:30, or until 4 clients reconnect
May 19 15:29:25 coss5 kernel: Lustre: charlie-OST0052: Will be in recovery for at least 2:30, or until 8 clients reconnect
May 19 15:29:25 coss5 kernel: Lustre: Skipped 4 previous similar messages
May 19 15:29:25 coss5 kernel: Lustre: Skipped 4 previous similar messages
May 19 15:31:36 coss5 kernel: Lustre: charlie-OST0044: recovery is timed out, evict stale exports
May 19 15:31:36 coss5 kernel: Lustre: charlie-OST0044: disconnecting 2 stale clients
May 19 15:31:36 coss5 kernel: Lustre: charlie-OST0047: Recovery over after 2:30, of 4 clients 2 recovered and 2 were evicted.
May 19 15:31:36 coss5 kernel: Lustre: charlie-OST004a: Recovery over after 2:30, of 4 clients 2 recovered and 2 were evicted.
May 19 15:31:37 coss5 kernel: LustreError: 15571:0:(ofd_obd.c:1339:ofd_create()) charlie-OST0049: unable to precreate: rc = -116
May 19 15:31:37 coss5 kernel: LustreError: 14051:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004a: unable to precreate: rc = -116
May 19 15:31:44 coss5 kernel: Lustre: charlie-OST004c: recovery is timed out, evict stale exports
May 19 15:31:44 coss5 kernel: Lustre: Skipped 7 previous similar messages
May 19 15:31:44 coss5 kernel: Lustre: charlie-OST004c: disconnecting 2 stale clients
May 19 15:31:44 coss5 kernel: Lustre: Skipped 7 previous similar messages
May 19 15:31:44 coss5 kernel: Lustre: charlie-OST004c: Recovery over after 2:30, of 4 clients 2 recovered and 2 were evicted.
May 19 15:31:44 coss5 kernel: Lustre: Skipped 6 previous similar messages
May 19 15:31:45 coss5 kernel: LustreError: 15233:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:31:46 coss5 kernel: LustreError: 14594:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004a: unable to precreate: rc = -116
May 19 15:31:54 coss5 kernel: LustreError: 14053:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:31:55 coss5 kernel: Lustre: charlie-OST0052: recovery is timed out, evict stale exports
May 19 15:31:55 coss5 kernel: Lustre: Skipped 5 previous similar messages
May 19 15:31:55 coss5 kernel: Lustre: charlie-OST0054: disconnecting 2 stale clients
May 19 15:31:55 coss5 kernel: Lustre: Skipped 5 previous similar messages
May 19 15:31:55 coss5 kernel: Lustre: charlie-OST0052: Recovery over after 2:30, of 8 clients 2 recovered and 6 were evicted.
May 19 15:31:55 coss5 kernel: Lustre: Skipped 5 previous similar messages
May 19 15:32:04 coss5 kernel: LustreError: 14053:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:32:04 coss5 kernel: LustreError: 14053:0:(ofd_obd.c:1339:ofd_create()) Skipped 1 previous similar message
May 19 15:32:14 coss5 kernel: LustreError: 14053:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:32:14 coss5 kernel: LustreError: 14053:0:(ofd_obd.c:1339:ofd_create()) Skipped 1 previous similar message
May 19 15:32:34 coss5 kernel: LustreError: 15233:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:32:34 coss5 kernel: LustreError: 15233:0:(ofd_obd.c:1339:ofd_create()) Skipped 3 previous similar messages
May 19 15:33:06 coss5 kernel: LustreError: 14051:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004a: unable to precreate: rc = -116
May 19 15:33:06 coss5 kernel: LustreError: 14051:0:(ofd_obd.c:1339:ofd_create()) Skipped 6 previous similar messages
May 19 15:34:14 coss5 kernel: LustreError: 15574:0:(ofd_obd.c:1339:ofd_create()) charlie-OST004f: unable to precreate: rc = -116
May 19 15:34:14 coss5 kernel: LustreError: 15574:0:(ofd_obd.c:1339:ofd_create()) Skipped 12 previous similar messages