https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5704 Lustre <p>lfs getstripe on fifo files hangs on 'open' system call. This worked in 2.1.x version.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mhanafi@pfe23:/nobackupp8/mhanafi&gt; rm testfifo mhanafi@pfe23:/nobackupp8/mhanafi&gt; mkfifo testfifo mhanafi@pfe23:/nobackupp8/mhanafi&gt; strace lfs getstripe testfifo execve("/usr/bin/lfs", ["lfs", "getstripe", "testfifo"], [/* 35 vars */]) = 0 brk(0) = 0x6c1000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fffedb02000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=270183, ...}) = 0 mmap(NULL, 270183, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fffedac0000 close(3) = 0 open("/lib64/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\200Z\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=135764, ...}) = 0 mmap(NULL, 2212784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fffed6c9000 fadvise64(3, 0, 2212784, POSIX_FADV_WILLNEED) = 0 mprotect(0x7fffed6e0000, 2097152, PROT_NONE) = 0 mmap(0x7fffed8e0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fffed8e0000 mmap(0x7fffed8e2000, 13232, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fffed8e2000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0P\355\1\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1775524, ...}) = 0 mmap(NULL, 3639480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fffed350000 fadvise64(3, 0, 3639480, POSIX_FADV_WILLNEED) = 0 mprotect(0x7fffed4c0000, 2093056, PROT_NONE) = 0 mmap(0x7fffed6bf000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16f000) = 0x7fffed6bf000 mmap(0x7fffed6c4000, 18616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fffed6c4000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fffedabf000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fffedabe000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fffedabd000 arch_prctl(ARCH_SET_FS, 0x7fffedabe700) = 0 mprotect(0x7fffed6bf000, 16384, PROT_READ) = 0 mprotect(0x7fffed8e0000, 4096, PROT_READ) = 0 mprotect(0x66a000, 4096, PROT_READ) = 0 mprotect(0x7fffedb04000, 4096, PROT_READ) = 0 munmap(0x7fffedac0000, 270183) = 0 set_tid_address(0x7fffedabe9d0) = 61077 set_robust_list(0x7fffedabe9e0, 0x18) = 0 futex(0x7fffffffe7dc, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0x7fffffffe7dc, 0x189 /* FUTEX_??? */, 1, NULL, 7fffedabe700) = -1 EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, {0x7fffed6ce8f0, [], SA_RESTORER|SA_SIGINFO, 0x7fffed6d8810}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x7fffed6ce980, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fffed6d8810}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=300000*1024, rlim_max=7340032*1024}) = 0 shmget(IPC_PRIVATE, 65680, 0600) = 438992901 shmat(438992901, 0, 0) = ? shmctl(438992901, IPC_RMID, 0) = 0 brk(0) = 0x6c1000 brk(0x6e2000) = 0x6e2000 open("testfifo", O_RDONLY^ </pre> </div></div> LU-5704

lfs getstripe hangs on fifo files

Bug Minor Resolved Fixed Bob Glossman Mahmoud Hanafi Thu, 2 Oct 2014 21:56:35 +0000 Fri, 16 Sep 2016 23:37:07 +0000 Sat, 2 Apr 2016 13:06:04 +0000 Lustre 2.4.3 Lustre 2.9.0 0 10 <p>Bob is trying to repro this issue</p> <p>This reproduces 100% in every version I have easy access to up through current master.</p> <p>I think this is expected behavior. It behaves the same way if the test fifo file is in a lustre filesystem or not. Looking at the kernel stack trace in crash shows that it is waiting in pipe_wait() in the kernel:</p> <p>PID: 110276 TASK: ffff88003a7b4300 CPU: 0 COMMAND: "lfs"<br/> #0 <span class="error">&#91;ffff88003a747b48&#93;</span> schedule at ffffffff81461beb<br/> #1 <span class="error">&#91;ffff88003a747c90&#93;</span> pipe_wait at ffffffff81164252<br/> #2 <span class="error">&#91;ffff88003a747ce0&#93;</span> fifo_open at ffffffff8116f4d6<br/> #3 <span class="error">&#91;ffff88003a747d10&#93;</span> __dentry_open at ffffffff81158f68<br/> #4 <span class="error">&#91;ffff88003a747d60&#93;</span> do_last at ffffffff811685c2<br/> #5 <span class="error">&#91;ffff88003a747dc0&#93;</span> path_openat at ffffffff81169839<br/> #6 <span class="error">&#91;ffff88003a747e50&#93;</span> do_filp_open at ffffffff81169cbc<br/> #7 <span class="error">&#91;ffff88003a747f20&#93;</span> do_sys_open at ffffffff8115a90f<br/> #8 <span class="error">&#91;ffff88003a747f80&#93;</span> system_call_fastpath at ffffffff8146bd12<br/> RIP: 00007f7848bb0fd0 RSP: 00007fff3f449fd8 RFLAGS: 00010206<br/> RAX: 0000000000000002 RBX: ffffffff8146bd12 RCX: 6f66696674736574<br/> RDX: 0000000000000050 RSI: 0000000000000000 RDI: 0000000000624030<br/> RBP: 000000000040fde0 R8: 00007fff3f44c447 R9: 0000000000000000<br/> R10: 00007fff3f449d80 R11: 0000000000000246 R12: 000000000040a320<br/> R13: 00007fff3f44a070 R14: 0000000000624030 R15: 00007fff3f44a000<br/> ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b</p> <p>I think it's normal behavior for a process that opens a fifo for read to just wait until somebody writes. I note if I echo a few chars into the test fifo file the lfs command wakes up and says:</p> <p>error: can't get lov name.: Invalid argument (22)<br/> testfifo has no stripe info</p> <p>then exits.</p> <p>I can't say why this ever worked in 2.1 unless the lfs code there somehow avoided doing an open on the file.</p> <p>Regardless of whether this ever worked, it's still a bug. This isn't "normal behavior" of any command that gets file attributes. I can do a stat on a fifo without hanging. I can do a getfattr on a fifo without hanging. I can do a getfacl on a fifo without hanging. Getting stripe info is no different than these. The implementation is flawed if it hangs and could be easily fixed with a fifo check that immediately prints the exact message shown after writing to the pipe. I can't imagine the fix would take more than a few minutes?!?</p> <p>the fact that getfattr/getfacl/stat on a fifo don't hang isn't relevant IMHO. you can do all those on a fifo in lustre too.</p> <p>lfs is a lustre specific tool. it does open on it's target(s) prior to doing anything else. I strongly disagree that it's a bug for it to hang in certain unlikely situations, like for instance trying to do lustre related stripe operations on files where that concept doesn't apply. a read of a fifo that nobody ever writes to will hang in precisely the same way.</p> <p>The reason that those other tools do not hang when opening a FIFO is because they pass open flags to prevent this.</p> <p>It appears that our <tt>open()</tt> call in <tt>llapi_file_get_stripe()</tt> is only passing <tt>O_RDONLY</tt> and it should also include <tt>O_NONBLOCK</tt> to avoid exactly this problem.</p> <p>'lfs getstripe' doesn't call llapi_file_get_stripe() at all. It does however do a different call sequence that has an open() call with only O_RDONLY in the flags.</p> <p>I can work up a patch to add O_NONBLOCK, but this particular call sequence is used for so many other things too I'm worried that doing so will cause bad side effects.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19039" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19039</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5704" title="lfs getstripe hangs on fifo files" class="issue-link" data-issue-key="LU-5704"><del>LU-5704</del></a> utils: stop open hangs on fifo files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bbb6139de7761d43a5b86e1ee14550d5f7ba2c02</p> <p>Bob, if you are worried that this patch "will cause bad side effects," we are not comfortable applying this patch to our production systems.</p> <p>Jay,<br/> At this point I believe my worry was unfounded. That comment was from before I created and submitted my patch. It has since undergone a complete set of review tests as well as passed inspection by reviewers.</p> <p>I think it's safe.</p> <p>Thanks for clarification, Bob.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19039/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19039/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5704" title="lfs getstripe hangs on fifo files" class="issue-link" data-issue-key="LU-5704"><del>LU-5704</del></a> utils: stop open hangs on fifo files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fab6073165dfbd107f54057e33363c77978af6cb</p> <p>Landed for 2.9</p> Related Development Rank 1|hzwxq7: Rank (Obsolete) 15976 Severity