Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-12459] parallel IOC_MDC_GETFILEINFO ioctl, getting ENOENT https://jira.whamcloud.com/browse/LU-12459 Lustre <p>When making a IOC_MDC_GETFILEINFO ioctl call across many threads in parallel on a client, I am randomly seeing ENOENT.</p> <p>Included lctl dk output.</p> LU-12459 parallel IOC_MDC_GETFILEINFO ioctl, getting ENOENT Bug Major Resolved Not a Bug Patrick Farrell Joe Grund Wed, 19 Jun 2019 20:22:13 +0000 Thu, 20 Jun 2019 17:22:29 +0000 Thu, 20 Jun 2019 17:22:29 +0000 0 4 <p>You're sure it's ENOPERM, and not ENOENT?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:00000001:3.0:1560975552.696763:0:20737:0:(module.c:119:libcfs_ioctl()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe) </pre> </div></div> <p>In the logs I see ENOENT, but not ENOPERM.</p> <p>Also, it looks like the root of the error is server side - Can you get dk logs from the MDS?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000001:3.0:1560975525.270337:0:20197:0:(client.c:2949:ptlrpc_queue_wait()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe) </pre> </div></div> <p>^ that's the server replying to the client, which is the first error recorded in the logs.</p> <p>Doing the same call serially from a single thread does not produce any errors.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pfarrell" class="user-hover" rel="pfarrell">pfarrell</a> Logs for MDS <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32826/32826_lctl_dk_mds.txt.zip" title="lctl_dk_mds.txt.zip attached to LU-12459">lctl_dk_mds.txt.zip<sup><img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span> .</p> <p>The errors appear to indicate Operation not permitted:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jun 19 20:35:08 c1.local iml-agent-daemon[7319]: [ERROR iml_agent::action_plugins::stratagem::action_warning] Failed to mdc_stat(/mnt/fs-client/foo99.txt) =&gt; Operation not permitted (os error 1)</pre> </div></div> <p>Hmm, are you able to strace one of the processes getting -1?  I ask because the -1 doesn't appear in the logs at all, so I can't see where it's coming from.  As far as I can see here, it doesn't appear to be the ioctl itself giving the failure...  (Could be wrong, strace will help either way).</p> <p>And, I guess, related:  Are you sure it's the ioctl() syscall specifically, and not, for example, the open before it (or anything else, really)?</p> <p>Good to know we might just be looking up things that don't exist, that's certainly the simplest explanation for the -2s in the logs.  (The MDS log does not obviously contain permission failures, I can dig a little, but I didn't want to wait to ask for the strace.)</p> <p>It's possible it could be outside the ioctl... I'll work on getting an strace.</p> <p>strace output for ioctl calls below. The way the error handling code was written on the IML side has a bug, which I'll look at fixing.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddec007b40) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddec020030) = -1 ENOENT (No such file or directory) ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007360) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory) ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde4039fa0) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory) ioctl(59, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde4001d90) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde4203020) = 0 ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd8003850) = 0 ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd8010a60) = -1 ENOENT (No such file or directory) ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc0041b0) = 0 ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc007980) = -1 ENOENT (No such file or directory) ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8002ef0) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc8001f00) = -1 ENOENT (No such file or directory) ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd00059a0) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0008c50) = 0 ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd40038b0) = 0 ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd4097040) = 0 ioctl(61, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc006e50) = 0 ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc121d20) = -1 ENOENT (No such file or directory) ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0003cf0) = 0 ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc0016e10) = -1 ENOENT (No such file or directory) ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc0a9720) = 0 ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc010680) = -1 ENOENT (No such file or directory) ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc40036d0) = 0 ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc40026e0) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4093a60) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = -1 ENOENT (No such file or directory) ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8006e50) = 0 ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb8005e60) = -1 ENOENT (No such file or directory) ioctl(53, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8004e30) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb8003e40) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac0023a0) = 0 ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac005890) = -1 ENOENT (No such file or directory) ioctl(62, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac004810) = 0 ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac003820) = 0 ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c002df0) = 0 ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c0966a0) = 0 ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8003480) = 0 ioctl(48, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8094570) = 0 ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda40054f0) = 0 ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4096530) = 0 ioctl(68, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4003290) = 0 ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4094fe0) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd98001e50) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd98000e60) = -1 ENOENT (No such file or directory) ioctl(54, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda0006390) = 0 ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda00a0440) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd900042a0) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900176a0) = -1 ENOENT (No such file or directory) ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd88093080) = 0 ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd88002060) = -1 ENOENT (No such file or directory) ioctl(50, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94005bf0) = 0 ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9400b120) = 0 ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c004610) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c007680) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780094f0) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd78135df0) = 0 ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780074d0) = 0 ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory) ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd80003f00) = 0 ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd80002f10) = 0 ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84007c30) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84138ba0) = 0 ioctl(51, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f908788cb0) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90881ab40) = 0 ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd78005470) = 0 ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd780a5de0) = 0 ioctl(47, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84004ff0) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84010b40) = -1 ENOENT (No such file or directory) ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c097690) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c005660) = -1 ENOENT (No such file or directory) ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94007e70) = 0 ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9412b140) = 0 ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9000f350) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900a76b0) = -1 ENOENT (No such file or directory) ioctl(60, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd980927f0) = 0 ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd98000e40) = -1 ENOENT (No such file or directory) ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda0004350) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda00073c0) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c002dd0) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c006690) = -1 ENOENT (No such file or directory) ioctl(43, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8003460) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8002470) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4003270) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4006520) = 0 ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddac004620) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddac0958a0) = -1 ENOENT (No such file or directory) ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8002dd0) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80080c0) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc0116b0) = 0 ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc0106c0) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4003a50) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = -1 ENOENT (No such file or directory) ioctl(64, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb4003a50) = 0 ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb4003a50) = 0 ioctl(33, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc409b740) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc40026c0) = -1 ENOENT (No such file or directory) ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0003d10) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc00a6e20) = 0 ioctl(56, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc0045c0) = 0 ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc091d10) = -1 ENOENT (No such file or directory) ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8005170) = 0 ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc8004180) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd0007c20) = 0 ioctl(44, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0006c30) = 0 ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd4005c70) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd4127050) = 0 ioctl(58, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc0041b0) = 0 ioctl(38, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc0d3150) = -1 ENOENT (No such file or directory) ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd8003850) = 0 ioctl(47, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd80a8ab0) = 0 ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d05c0) = 0 ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory) ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007340) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory) ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde0007380) = 0 ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde0008650) = -1 ENOENT (No such file or directory) ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddec01f000) = 0 ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddec0b0040) = -1 ENOENT (No such file or directory) ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde8001a80) = 0 ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde8000a90) = 0 ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf0002880) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf0094b40) = 0 ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf8001400) = 0 ioctl(44, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf8006520) = -1 ENOENT (No such file or directory) ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf4004160) = 0 ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4095400) = -1 ENOENT (No such file or directory) ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf40042a0) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4007430) = 0 ioctl(46, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f908788c90) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90878a2c0) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780074b0) = 0 ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory) ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd78008500) = 0 ioctl(49, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory) ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84004ff0) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd84010b40) = -1 ENOENT (No such file or directory) ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd90006780) = -1 ENODATA (No data available) ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd900062e0) = 0 ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd900a76b0) = 0 ioctl(57, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda009f410) = 0 ioctl(30, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda000d3e0) = -1 ENOENT (No such file or directory) ioctl(38, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9c0966a0) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9c006690) = -1 ENOENT (No such file or directory) ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8004e10) = 0 ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80980d0) = -1 ENOENT (No such file or directory) ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddb8002df0) = 0 ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddb80060a0) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddbc011690) = 0 ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddbc0106a0) = 0 ioctl(31, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc0005d50) = 0 ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc0016e10) = -1 ENOENT (No such file or directory) ioctl(34, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddcc0045c0) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddcc091d10) = -1 ENOENT (No such file or directory) ioctl(35, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddc8002ed0) = 0 ioctl(37, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddc80063e0) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd0003720) = 0 ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd0008c50) = 0 ioctl(55, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd4007cb0) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd41b7060) = 0 ioctl(66, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdddc008990) = 0 ioctl(39, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdddc0079a0) = -1 ENOENT (No such file or directory) ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d05c0) = 0 ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde40e3000) = -1 ENOENT (No such file or directory) ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdde40d2600) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdde4173010) = 0 ioctl(48, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddd800c5a0) = 0 ioctl(28, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddd8010a60) = -1 ENOENT (No such file or directory) ioctl(69, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x55f90878bb20) = 0 ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x55f90878ab30) = 0 ioctl(17, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf40063e0) = 0 ioctl(18, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf40053f0) = 0 ioctl(67, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf4006400) = 0 ioctl(25, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf4097440) = -1 ENOENT (No such file or directory) ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf00027e0) = 0 ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf00017f0) = 0 ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fddf0004b20) = 0 ioctl(45, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fddf0004b30) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd780064c0) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd7800dd90) = -1 ENOENT (No such file or directory) ioctl(36, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd80001ec0) = 0 ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd80005170) = 0 ioctl(41, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd84009c70) = 0 ioctl(42, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd840a8b90) = 0 ioctl(29, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd88003050) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd88002060) = 0 ioctl(40, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd880930a0) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd880920b0) = 0 ioctl(19, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c003600) = 0 ioctl(20, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c003600) = -1 ENOENT (No such file or directory) ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd8c006650) = 0 ioctl(26, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd8c0a0700) = 0 ioctl(27, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd94005990) = 0 ioctl(32, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd940049a0) = 0 ioctl(52, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdd9400a0f0) = 0 ioctl(22, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdd9409b130) = 0 ioctl(63, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda8005550) = 0 ioctl(21, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda8004560) = -1 ENOENT (No such file or directory) ioctl(65, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x96, 0x08), 0x7fdda4005fc0) = 0 ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x08), 0x7fdda4004fd0) = 0 </pre> </div></div> <p>OK, so just ENOENT then.  I'm guessing you had confusion between ERRNO and return code, then.  Thanks for checking.  I'll keep an eye on the other ticket, but it seems likely it's all one cause.  We can keep this open until we're sure.</p> <p> </p> <p>Updated IML error that is now correctly reflecting the error code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Failed to mdc_stat(/mnt/fs-client/foo84.txt) =&gt; No such file or directory (os error 2) </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pfarrell" class="user-hover" rel="pfarrell">pfarrell</a> I'm not sure it is the same cause as these calls are for files that do actually exist on the client, the other ticket references fids that do not exist on the client.</p> <p>Ah, OK - So you are filtering and/or have tested the FIDs for which this call failed via some other method?</p> <p>yes, all these files have already run through fid2path to get their path directly before the ioctl call. I also can look in the client dir and see they are there. Finally, doing this same thing serially (from a single thread), there are no errors produced.</p> <p>@Joe Grund: Which version of lustre is affected by this problem?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lflis" class="user-hover" rel="lflis">lflis</a> I'm seeing this against Lustre 2.12.1</p> <p>Two questions:<br/> 1. What was the debug mask used?  -1 ?<br/> 2. What's the exact Lustre version here (client &amp; server)?</p> <p>Not sure what the debug mask is but I did do <tt>lctl set_param debug=+trace</tt>.</p> <p>I am running lustre 2.12.1 on all servers and clients.</p> <p>Sorry for the noise, this one is an issue with IML code. Needed to ensure NUL byte was getting copied over for the ioctl call.</p> Development Rank 1|i00iif: Rank (Obsolete) 9223372036854775807 Severity