[LU-12459] parallel IOC_MDC_GETFILEINFO ioctl, getting ENOENT

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Major
• Reporter: Joe Grund
• Assignee: Patrick Farrell (Inactive)
• Resolution: Not a Bug
• Severity: 3

Description

When making a IOC_MDC_GETFILEINFO ioctl call across many threads in parallel on a client, I am randomly seeing ENOENT.

Included lctl dk output.

Comments

[Patrick Farrell (Inactive)]

You're sure it's ENOPERM, and not ENOENT?

00000400:00000001:3.0:1560975552.696763:0:20737:0:(module.c:119:libcfs_ioctl()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe) 

In the logs I see ENOENT, but not ENOPERM.

Also, it looks like the root of the error is server side - Can you get dk logs from the MDS?

00000100:00000001:3.0:1560975525.270337:0:20197:0:(client.c:2949:ptlrpc_queue_wait()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe) 

^ that's the server replying to the client, which is the first error recorded in the logs.

[Joe Grund]

Doing the same call serially from a single thread does not produce any errors.

[Joe Grund]

pfarrell Logs for MDS lctl_dk_mds.txt.zip .

The errors appear to indicate Operation not permitted:

Jun 19 20:35:08 c1.local iml-agent-daemon[7319]: [ERROR iml_agent::action_plugins::stratagem::action_warning] Failed to mdc_stat(/mnt/fs-client/foo99.txt) => Operation not permitted (os error 1)

[Patrick Farrell (Inactive)]

Hmm, are you able to strace one of the processes getting -1?  I ask because the -1 doesn't appear in the logs at all, so I can't see where it's coming from.  As far as I can see here, it doesn't appear to be the ioctl itself giving the failure...  (Could be wrong, strace will help either way).

And, I guess, related:  Are you sure it's the ioctl() syscall specifically, and not, for example, the open before it (or anything else, really)?

Good to know we might just be looking up things that don't exist, that's certainly the simplest explanation for the -2s in the logs.  (The MDS log does not obviously contain permission failures, I can dig a little, but I didn't want to wait to ask for the strace.)

[Joe Grund]

It's possible it could be outside the ioctl... I'll work on getting an strace.

[Joe Grund]

strace output for ioctl calls below. The way the error handling code was written on the IML side has a bug, which I'll look at fixing.

ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddec007b40) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddec020030) = -1 ENOENT (No such file or directory)
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007360) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory)
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde4039fa0) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory)
ioctl(59, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde4001d90) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde4203020) = 0
ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd8003850) = 0
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd8010a60) = -1 ENOENT (No such file or directory)
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc0041b0) = 0
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc007980) = -1 ENOENT (No such file or directory)
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8002ef0) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc8001f00) = -1 ENOENT (No such file or directory)
ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd00059a0) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0008c50) = 0
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd40038b0) = 0
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd4097040) = 0
ioctl(61, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc006e50) = 0
ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc121d20) = -1 ENOENT (No such file or directory)
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0003cf0) = 0
ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc0016e10) = -1 ENOENT (No such file or directory)
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc0a9720) = 0
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc010680) = -1 ENOENT (No such file or directory)
ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc40036d0) = 0
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc40026e0) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4093a60) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = -1 ENOENT (No such file or directory)
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8006e50) = 0
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb8005e60) = -1 ENOENT (No such file or directory)
ioctl(53, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8004e30) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb8003e40) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac0023a0) = 0
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac005890) = -1 ENOENT (No such file or directory)
ioctl(62, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac004810) = 0
ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac003820) = 0
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c002df0) = 0
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c0966a0) = 0
ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8003480) = 0
ioctl(48, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8094570) = 0
ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda40054f0) = 0
ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4096530) = 0
ioctl(68, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4003290) = 0
ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4094fe0) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd98001e50) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd98000e60) = -1 ENOENT (No such file or directory)
ioctl(54, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda0006390) = 0
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda00a0440) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd900042a0) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900176a0) = -1 ENOENT (No such file or directory)
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd88093080) = 0
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd88002060) = -1 ENOENT (No such file or directory)
ioctl(50, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94005bf0) = 0
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9400b120) = 0
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c004610) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c007680) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780094f0) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd78135df0) = 0
ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780074d0) = 0
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory)
ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd80003f00) = 0
ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd80002f10) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84007c30) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84138ba0) = 0
ioctl(51, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f908788cb0) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90881ab40) = 0
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd78005470) = 0
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd780a5de0) = 0
ioctl(47, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84004ff0) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84010b40) = -1 ENOENT (No such file or directory)
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c097690) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c005660) = -1 ENOENT (No such file or directory)
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94007e70) = 0
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9412b140) = 0
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9000f350) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900a76b0) = -1 ENOENT (No such file or directory)
ioctl(60, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd980927f0) = 0
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd98000e40) = -1 ENOENT (No such file or directory)
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda0004350) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda00073c0) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c002dd0) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c006690) = -1 ENOENT (No such file or directory)
ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8003460) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8002470) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4003270) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4006520) = 0
ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac004620) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac0958a0) = -1 ENOENT (No such file or directory)
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8002dd0) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80080c0) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc0116b0) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc0106c0) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4003a50) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = -1 ENOENT (No such file or directory)
ioctl(64, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4003a50) = 0
ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = 0
ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc409b740) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc40026c0) = -1 ENOENT (No such file or directory)
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0003d10) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc00a6e20) = 0
ioctl(56, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc0045c0) = 0
ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc091d10) = -1 ENOENT (No such file or directory)
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8005170) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc8004180) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd0007c20) = 0
ioctl(44, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0006c30) = 0
ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd4005c70) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd4127050) = 0
ioctl(58, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc0041b0) = 0
ioctl(38, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc0d3150) = -1 ENOENT (No such file or directory)
ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd8003850) = 0
ioctl(47, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd80a8ab0) = 0
ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d05c0) = 0
ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory)
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007340) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory)
ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007380) = 0
ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory)
ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddec01f000) = 0
ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddec0b0040) = -1 ENOENT (No such file or directory)
ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde8001a80) = 0
ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde8000a90) = 0
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf0002880) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf0094b40) = 0
ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf8001400) = 0
ioctl(44, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf8006520) = -1 ENOENT (No such file or directory)
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf4004160) = 0
ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4095400) = -1 ENOENT (No such file or directory)
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf40042a0) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4007430) = 0
ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f908788c90) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90878a2c0) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780074b0) = 0
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory)
ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd78008500) = 0
ioctl(49, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory)
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84004ff0) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84010b40) = -1 ENOENT (No such file or directory)
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd90006780) = -1 ENODATA (No data available)
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd900062e0) = 0
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900a76b0) = 0
ioctl(57, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda009f410) = 0
ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda000d3e0) = -1 ENOENT (No such file or directory)
ioctl(38, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c0966a0) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c006690) = -1 ENOENT (No such file or directory)
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8004e10) = 0
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80980d0) = -1 ENOENT (No such file or directory)
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8002df0) = 0
ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80060a0) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc011690) = 0
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc0106a0) = 0
ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0005d50) = 0
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc0016e10) = -1 ENOENT (No such file or directory)
ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc0045c0) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc091d10) = -1 ENOENT (No such file or directory)
ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8002ed0) = 0
ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc80063e0) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd0003720) = 0
ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0008c50) = 0
ioctl(55, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd4007cb0) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd41b7060) = 0
ioctl(66, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc008990) = 0
ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc0079a0) = -1 ENOENT (No such file or directory)
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d05c0) = 0
ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory)
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d2600) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde4173010) = 0
ioctl(48, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd800c5a0) = 0
ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd8010a60) = -1 ENOENT (No such file or directory)
ioctl(69, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f90878bb20) = 0
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90878ab30) = 0
ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf40063e0) = 0
ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf40053f0) = 0
ioctl(67, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf4006400) = 0
ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4097440) = -1 ENOENT (No such file or directory)
ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf00027e0) = 0
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf00017f0) = 0
ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf0004b20) = 0
ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf0004b30) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780064c0) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory)
ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd80001ec0) = 0
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd80005170) = 0
ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84009c70) = 0
ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd840a8b90) = 0
ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd88003050) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd88002060) = 0
ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd880930a0) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd880920b0) = 0
ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c003600) = 0
ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c003600) = -1 ENOENT (No such file or directory)
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c006650) = 0
ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c0a0700) = 0
ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94005990) = 0
ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd940049a0) = 0
ioctl(52, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9400a0f0) = 0
ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9409b130) = 0
ioctl(63, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8005550) = 0
ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8004560) = -1 ENOENT (No such file or directory)
ioctl(65, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4005fc0) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4004fd0) = 0

[Patrick Farrell (Inactive)]

OK, so just ENOENT then.  I'm guessing you had confusion between ERRNO and return code, then.  Thanks for checking.  I'll keep an eye on the other ticket, but it seems likely it's all one cause.  We can keep this open until we're sure.

 

[Joe Grund]

Updated IML error that is now correctly reflecting the error code:

Failed to mdc_stat(/mnt/fs-client/foo84.txt) => No such file or directory (os error 2)

[Joe Grund]

pfarrell I'm not sure it is the same cause as these calls are for files that do actually exist on the client, the other ticket references fids that do not exist on the client.

[Patrick Farrell (Inactive)]

Ah, OK - So you are filtering and/or have tested the FIDs for which this call failed via some other method?

[Joe Grund]

yes, all these files have already run through fid2path to get their path directly before the ioctl call. I also can look in the client dir and see they are there. Finally, doing this same thing serially (from a single thread), there are no errors produced.

[Lukasz Flis]

@Joe Grund: Which version of lustre is affected by this problem?

[Joe Grund]

lflis I'm seeing this against Lustre 2.12.1

[Patrick Farrell (Inactive)]

Two questions:
1. What was the debug mask used?  -1 ?
2. What's the exact Lustre version here (client & server)?

[Joe Grund]

Not sure what the debug mask is but I did do lctl set_param debug=+trace.

I am running lustre 2.12.1 on all servers and clients.

[Joe Grund]

Sorry for the noise, this one is an issue with IML code. Needed to ensure NUL byte was getting copied over for the ioctl call.