https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16808 Lustre <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ mkfifo myfifo $ lfs find myfifo myfifo $ lfs find myfifo --printf <span class="code-quote">"%p\n"</span> ** command hangs ** ^C $ sudo mknod mychardev c 1 5 $ lfs find mychardev mychardev $ lfs find mychardev --printf <span class="code-quote">"%p\n"</span> lfs: failed <span class="code-keyword">for</span> <span class="code-quote">'mychardev'</span>: Inappropriate ioctl <span class="code-keyword">for</span> device</pre> </div></div> LU-16808

lfs find --printf fails on FIFOs and special files

Bug Minor Resolved Fixed Zhenyu Xu Thomas Bertschinger Tue, 9 May 2023 14:33:38 +0000 Tue, 20 Jun 2023 17:23:54 +0000 Tue, 20 Jun 2023 17:23:54 +0000 Lustre 2.16.0 0 5 <p>So it shouldn't hang on the fifo obviously, but can you explain the behavior you'd expect on these special files?  I mean, what does lfs find do in general if called on a regular file rather than a directory?</p> <p>I suppose I'd expect the printf to print out the desired information without error (as long as there are no Lustre-specific format directives that don't apply to special files). I think this would mainly be relevant if you pass `lfs find` a directory and there are special files under that directory tree. I guess it's probably not common to put special files on Lustre so I'll admit this likely wouldn't be seen on real systems.</p> <p>I would expect this to work without error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ cd /mnt/lustre $ touch a $ mknod mychardev c 1 5 $ lfs find /mnt/lustre --printf <span class="code-quote">"%p\n"</span> /mnt/lustre /mnt/lustre/a lfs: failed <span class="code-keyword">for</span> <span class="code-quote">'/mnt/lustre'</span>: Inappropriate ioctl <span class="code-keyword">for</span> device </pre> </div></div> <p>But this is probably fine failing (unless maybe the %Li should just be empty or have a placeholder value in the line for the special file?):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ $ lfs find /mnt/lustre --printf <span class="code-quote">"%Li %p\n"</span> 0 /mnt/lustre 1 /mnt/lustre/a lfs: failed <span class="code-keyword">for</span> <span class="code-quote">'/mnt/lustre'</span>: Inappropriate ioctl <span class="code-keyword">for</span> device</pre> </div></div> <p>Also, I think that error message should include the filename that failed because if you're running find on a big tree it won't necessarily be obvious what failed.</p> <p>"I guess it's probably not common to put special files on Lustre so I'll admit this likely wouldn't be seen on real systems."<br/> Well, people definitely do it, and special files can also be an implementation detail of some libraries/program environments/build systems/etc, so they do matter for us.  I agree it's not a massive gap, but it's a valid concern.</p> <p>FWIW, after reading your response, I think in general we should probably not return inappropriate ioctl (and obviously not for the requests which have a clear correct answer) - we should return whatever makes the most sense.  imo, it's confusing to get an error in the middle of an fs scan type operation.</p> <p>Glancing in to the code, it looks like our ll_special_inode_operations (our inode operations for all file structs which are not directories, regular files, or links) does not provide the .unlocked_ioctl function, so Lustre has no support for Lustre specific ioctls on special files.  (I'm just going to use 'special files' to mean anything that's not a directory, regular file, or symlink.)</p> <p>I think you have opened a bit of a can of worms here, unfortunately, since now you may have to consider which Lustre specific ioctls (and there are a <b>lot</b> of them) are meaningful on special files and give them implementations.  (Possibly it might make sense to share the ll_file_ioctl function as used for regular files, but you'd still have to walk through all the ioctls and add various checks.)</p> <p>Oh and we don't provide unlocked_ioctl for symlinks either!  That's interesting; I wonder how that works in practice and how others/VFS handles this.</p> <p>Like I said, a little bit of a can of worms.</p> <p>It's probably possible to figure out what ioctls 'find' is using and just make sure those have handling for special files (and possibly symlinks too, as appropriate?), and ignore the rest of the Lustre specific ioctls.  That wouldn't be a regression from current behavior, where no Lustre specific ioctls work on special files at all (afaics).</p> <p>Oh, and absolutely about the error message including the file name, that's obviously right.  That could be a quick separate patch if you wanted to do it.</p> <p>Also, I didn't look at why find hangs on fifos, that's probably the most problematic issue here.</p> <p>The FIFO hang is probably because the <tt>open()</tt> immediately gets blocked on waiting for some data to be available on the pipe? The strace shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>write(1, "/mnt/testfs/newdir\n", 19/mnt/testfs/newdir ) = 19 close(3) = 0 fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 fcntl(4, F_GETFL) = 0x28800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW) fcntl(4, F_SETFD, FD_CLOEXEC) = 0 getdents(4, /* 3 entries */, 32768) = 72 ioctl(4, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x140), 0x16f3040) = 0 ioctl(4, _IOC(_IOC_READ, 0x66, 0xaf, 0x4), 0x7ffd455d7480) = 0 open("/mnt/testfs/newdir/char", O_RDONLY) = 3 ioctl(3, FS_IOC_FSGETXATTR, 0x7ffd455d70b0) = -1 ENOTTY (Inappropriate ioctl for device) close(3) = 0 ioctl(4, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x140), 0x1f84040) = 0 ioctl(4, _IOC(_IOC_READ, 0x66, 0xaf, 0x4), 0x7ffc5a629f30) = 0 open("/mnt/testfs/newdir/fifo", O_RDONLY </pre> </div></div> <p>I suspect that the "semantic traverse" can already use the file type in the directory entry to skip trying to process such files.</p> <p>In fact, looking at the strace output for regular file processing it seems like it is doing far too much for most files already:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1683657120.708615 getdents(4, /* 102 entries */, 32768) = 6448 &lt;0.000784&gt; 1683657120.709513 ioctl(4, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x140), 0xea2040) = 0 &lt;0.002730&gt; 1683657120.712309 open("/mnt/testfs/dir/filename_is_long-15", O_RDONLY) = 3 &lt;0.003517&gt; 1683657120.715904 ioctl(3, _IOC(_IOC_READ, 0x66, 0xaf, 0x4), 0x7ffc0a9b3ba0) = 0 &lt;0.000140&gt; 1683657120.716106 ioctl(3, FS_IOC_FSGETXATTR, 0x7ffc0a9b37d0) = 0 &lt;0.000044&gt; 1683657120.716243 newfstatat(4, "filename_is_long-15", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 &lt;0.000683&gt; 1683657120.717022 write(1, "/mnt/testfs/dir/filename_is_long-15", 36) = 36 &lt;0.000296&gt; /mnt/testfs/dir/filename_is_long-15 1683657120.717386 close(3) = 0 &lt;0.002073&gt; </pre> </div></div> <p>The <tt>ioctl(4, _IOC(_IOC_READ|_IOC_WRITE, 0x69, 0x16, 0x140) = IOC_MDC_GETFILEINFO_V2)</tt> is expected for every file, since it gets the file layout and MDT attributes <b>from the parent directory</b> but we definitely <b>should not</b> be doing <tt>open()</tt> and <tt>ioctl(FS_IOC_FSGETXATTR)</tt> and <tt>newfstatat()</tt> on <b>every</b> file when we are only printing the pathname. That is a <b>lot</b> of overhead (more than double the original <tt>ioctl(IOC_MDC_GETFILEINFO_V2)</tt> call.</p> <p>It looks like this was added as part of the <tt>-printf</tt> functionality, since running without <tt>-printf "%p"</tt> shows none of that overhead:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1683657295.265889 getdents(4, /* 102 entries */, 32768) = 6448 &lt;0.000393&gt; 1683657295.266362 write(1, "/mnt/testfs/dir/filename_is_long-15"..., 36) = 36 &lt;0.000009&gt; /mnt/testfs/dir/filename_is_long-15 1683657295.266446 write(1, "/mnt/testfs/dir/filename_is_long-44"..., 36) = 36 &lt;0.000009&gt; /mnt/testfs/dir/filename_is_long-44 </pre> </div></div> <p>I guess that is already described in <a href="https://jira.whamcloud.com/browse/LU-15837" title="&quot;lfs find -printf&quot; improvements" class="issue-link" data-issue-key="LU-15837">LU-15837</a> under "<tt>selective fetching of metadata attributes. In particular, fetching the projid for a file requires an extra syscall that is unnecessary if the projid is not being printed</tt>" but it isn't just one syscall but multiple.</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50905</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16808" title="lfs find --printf fails on FIFOs and special files" class="issue-link" data-issue-key="LU-16808"><del>LU-16808</del></a> lfs: lfs find --printf won't hang on special files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 935a7716d744d2c2009a363f0b1bdbaee84d0895</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50916" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50916</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16808" title="lfs find --printf fails on FIFOs and special files" class="issue-link" data-issue-key="LU-16808"><del>LU-16808</del></a> lfs: return error if projid cannot be retrieved<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 37871eccbaafe1e2b7a07dd74070ef7ba2d40751</p> <p>I noticed another (potential) issue with symlinks and the project ID. For most attributes it appears <tt>lfs find</tt> uses data from the symlink itself, but for project ID, <tt>lfs find</tt> seems to use the value from the target:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ touch file20 &amp;&amp; lfs project -p 20 file20 $ ln -s file20 link21 &amp;&amp; lfs project -p 21 link21 $ lfs project .    20 - ./file20    21 - ./link21 # including %s in the format string to demonstrate that lfs find uses values from the link itself (and not the target) <span class="code-keyword">for</span> attributes other than project id $ lfs find . --printf <span class="code-quote">"%p %s %LP\n"</span> . 4096 0 ./file20 0 20 ./link21 6 20</pre> </div></div> <p>Would we expect that <tt>lfs find</tt> would use the value from the link in this case?</p> <p>Interestingly, I ran a quick test on ext4 and found that (at least in my setup) projid on symlinks did not work:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ext4-mount $ touch file20 &amp;&amp; chattr -p 20 file20 ext4-mount $ ln -s file20 link21 ext4-mount $ chattr -p 21 link21 chattr: Operation not supported <span class="code-keyword">while</span> reading flags on link21 ext4-mount $ lsattr -p .     0 --------------e------- ./lost+found lsattr: Operation not supported While reading flags on ./link21    20 --------------e------- ./file20</pre> </div></div> <p>I am not sure what the "expected" behavior of projid and symlinks is so I don't know if the <tt>lfs find</tt> behavior above is problematic.</p> <p>@adilger any thoughts on this? If the current behavior is wrong should I submit a ticket for this?</p> <p>Getting the projid on the symlink target is intentional behavior (<a href="https://jira.whamcloud.com/browse/LU-11872" title="Request for option not to follow symlink when setting project ID" class="issue-link" data-issue-key="LU-11872"><del>LU-11872</del></a>).</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50905/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50905/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16808" title="lfs find --printf fails on FIFOs and special files" class="issue-link" data-issue-key="LU-16808"><del>LU-16808</del></a> lfs: lfs find --printf won't hang on special files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f47f816a5b1deabc1d501332a1a30af205d22515</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50916/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50916/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16808" title="lfs find --printf fails on FIFOs and special files" class="issue-link" data-issue-key="LU-16808"><del>LU-16808</del></a> lfs: return invalid=-1 if no projid retrieved<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8d09b3e77744973d10d2029028d7e6f14591bc6d</p> <p>Landed for 2.16</p> Related LU-11872 LU-15837 Development Rank 1|i03kv3: Rank (Obsolete) 9223372036854775807 Severity