https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11069 Lustre <p>On lustre2.10.3 ifort lseek will return the wrong position when using append "&gt;&gt;" operation.</p> <p>This can be reproduced on</p> <p>client: sles12 lustre2.103 &lt;–&gt; server: lustre2.7.3</p> <p>client: sles12 lustre2.103 &lt;–&gt; server: lustre2.10.3</p> <p>hello.f</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> write(6,*) <span class="code-quote">"hello world"</span> end </pre> </div></div> <p>Working on sles11 and lustre2.7.3</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&gt; date &gt; tt &gt; date &gt;&gt; tt &gt; cat tt Wed May 30 13:32:58 PDT 2018 Wed May 30 13:33:04 PDT 2018 &gt; strace ./a.out &gt;&gt; tt ..... ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffffffbf80) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) fstat(1, {st_mode=S_IFREG|0600, st_size=58, ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffffffbf80) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) write(1, <span class="code-quote">" hello world\n"</span>, 13) = 13 lseek(1, 0, SEEK_CUR) = 71 ftruncate(1, 71) = 0 &gt; cat tt Wed May 30 13:32:58 PDT 2018 Wed May 30 13:33:04 PDT 2018 hello world </pre> </div></div> <ul> <li> <ul> <li>Note lseek(1,0,SEEK_CUR) set at 71.</li> </ul> </li> </ul> <p>Not working on lustre2.10.3.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&gt; rm tt &gt; date &gt; tt &gt; date &gt;&gt; tt &gt; cat tt Wed May 30 13:41:39 PDT 2018 Wed May 30 13:41:44 PDT 2018 &gt; strace ./a.out &gt;&gt; tt .... fstat(1, {st_mode=S_IFREG|0600, st_size=58, ...}) = 0 ioctl(1, TCGETS, 0x7fffffffbf80) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) write(1, <span class="code-quote">" hello world\n"</span>, 13) = 13 lseek(1, 0, SEEK_CUR) = 13 ftruncate(1, 13) = 0 pfe27:/nobackup/jchang&gt; cat tt Wed May 30 13pfe27:/nobackup/jchang&gt; </pre> </div></div> <p>*<b>Note that it's lseek that is getting the **wrong</b>* information about current position after the append. </p> sles12 lustre 2.10.3 LU-11069

ifort lseek returns wrong position on lustre 2.10.3

Bug Critical Resolved Fixed John Hammond Mahmoud Hanafi Fri, 1 Jun 2018 22:18:42 +0000 Wed, 15 May 2019 02:28:50 +0000 Tue, 19 Jun 2018 20:07:39 +0000 Lustre 2.12.0 Lustre 2.10.3 Lustre 2.12.0 Lustre 2.10.5 0 12 <p>Looks similar to <a href="https://jira.whamcloud.com/browse/LU-3044" title="LSeek SEEK_CUR gives incorrect value after write when file is open with O_APPEND" class="issue-link" data-issue-key="LU-3044"><del>LU-3044</del></a> (LSeek SEEK_CUR gives incorrect value after write when file is open with O_APPEND), which was fixed... but perhaps there was a regression in a similar location in the code.</p> <p>This is not limited to ifort and this is on master as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>n:~# cat test.c #include &lt;unistd.h&gt; int main(int argc, char *argv[]) { write(1, "Hello world!\n", 13); lseek(1, 0, SEEK_CUR); ftruncate(1, 13); return 0; } n:~# gcc -Wall test.c -o test n:~# cd /mnt/lustre n:lustre# rm -f f0 n:lustre# date &gt;&gt; f0 n:lustre# date &gt;&gt; f0 n:lustre# cat f0 Sat Jun 2 12:20:11 CDT 2018 Sat Jun 2 12:20:12 CDT 2018 n:lustre# strace ~/test &gt;&gt; f0 execve("/root/test", ["/root/test"], [/* 36 vars */]) = 0 brk(NULL) = 0x1c4f000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd6506da000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=46571, ...}) = 0 mmap(NULL, 46571, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd6506ce000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=2127336, ...}) = 0 mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd6500f7000 mprotect(0x7fd6502af000, 2097152, PROT_NONE) = 0 mmap(0x7fd6504af000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x7fd6504af000 mmap(0x7fd6504b5000, 16832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd6504b5000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd6506cd000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd6506cb000 arch_prctl(ARCH_SET_FS, 0x7fd6506cb740) = 0 mprotect(0x7fd6504af000, 16384, PROT_READ) = 0 mprotect(0x600000, 4096, PROT_READ) = 0 mprotect(0x7fd6506db000, 4096, PROT_READ) = 0 munmap(0x7fd6506ce000, 46571) = 0 write(1, "Hello world!\n", 13) = 13 lseek(1, 0, SEEK_CUR) = 13 ftruncate(1, 13) = 0 exit_group(0) = ? +++ exited with 0 +++ n:lustre# cat f0 Sat Jun 2 12n:lustre# </pre> </div></div> <p>Note that the same behaviour with John's test program is seen on both HFS+ and ext4, so it isn't clear if this is really a Lustre bug.</p> <p>Mahmoud, can you please attach your original test program, since I don't think John's is exercising this problem correctly.</p> <p>&gt; Note that the same behaviour with John's test program is seen on both HFS+ and ext4, so it isn't clear if this is really a Lustre bug.</p> <p>When I run my test program on ext4 (3.10.0-693.11.6.el7.lustre.x86_64) I see the correct behavior:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>n:~# cd /tmp n:tmp# rm -f f0 n:tmp# date &gt;&gt; f0 n:tmp# date &gt;&gt; f0 n:tmp# cat f0 Mon Jun 4 08:28:09 CDT 2018 Mon Jun 4 08:28:10 CDT 2018 n:tmp# strace ~/test &gt;&gt; f0 execve("/root/test", ["/root/test"], [/* 36 vars */]) = 0 brk(NULL) = 0x1ccc000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff77014e000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=46571, ...}) = 0 mmap(NULL, 46571, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff770142000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=2127336, ...}) = 0 mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff76fb6b000 mprotect(0x7ff76fd23000, 2097152, PROT_NONE) = 0 mmap(0x7ff76ff23000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x7ff76ff23000 mmap(0x7ff76ff29000, 16832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff76ff29000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff770141000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff77013f000 arch_prctl(ARCH_SET_FS, 0x7ff77013f740) = 0 mprotect(0x7ff76ff23000, 16384, PROT_READ) = 0 mprotect(0x600000, 4096, PROT_READ) = 0 mprotect(0x7ff77014f000, 4096, PROT_READ) = 0 munmap(0x7ff770142000, 46571) = 0 write(1, "Hello world!\n", 13) = 13 lseek(1, 0, SEEK_CUR) = 71 ftruncate(1, 13) = 0 exit_group(0) = ? +++ exited with 0 +++ </pre> </div></div> <p>Note that <tt>lseek(1, 0, SEEK_CUR)</tt> returns 71 (which is correct) rather than 13 as on Lustre.</p> <p>Andreas, the fortran reproducer is the two line "hello.f" in the original description.</p> <p>Bisection shows that this was introduced by <a href="https://review.whamcloud.com/26468" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26468</a> <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> clio: Parallelize generic I/O.</p> <p>My confusion is that I was looking at the file output at the end, which was always truncated, due to the test program calling <tt>ftruncate(1, 13)</tt> each time. Was that supposed to be <tt>ftruncate(1, &lt;lseek_return&gt;)</tt> or similar? In any case, just checking the return of <tt>lseek()</tt> shows that this is indeed a Lustre-specific issue. It would be good to make the test tool check the file size before and after the append and make that clear. </p> <p>It seems possible to reproduce this problem with <tt>multiop</tt> instead of making a dedicated test binary for the regression test.</p> <p>Can we expect a patch soon? Or should we pull the offending patch, in our build, until a fix is available.</p> <p>Hi Mahmoud,</p> <p>I have a patch which passes my version of your test case. (I expect that it will pass yours as well.) I am validating it now.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32641" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32641</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11069" title="ifort lseek returns wrong position on lustre 2.10.3" class="issue-link" data-issue-key="LU-11069"><del>LU-11069</del></a> llite: correct file position after appending writes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: adb493be610960871720353fdc51b231dd38f976</p> <p>I have a conflict applying this patch to 2.10.3 at lustre/tests/multiop.c:</p> <p>int main(int argc, char **argv)<br/> {<br/> char *fname, *commands;<br/> const char *newfile;<br/> const char *oldpath;<br/> struct stat st;<br/> struct statfs stfs;<br/> size_t mmap_len = 0, i;<br/> unsigned char *mmap_ptr = NULL, junk = 0;<br/> int len, fd = -1;<br/> int flags;<br/> int save_errno;<br/> int verbose = 0;<br/> int gid = 0;<br/> lustre_fid fid;<br/> struct timespec ts;<br/> struct lov_user_md_v3 lum;<br/> &lt;&lt;&lt;&lt;&lt;&lt;&lt; HEAD<br/> __u64 dv;<br/> =======<br/> long long rc = 0;<br/> long long last_rc;<br/> &gt;&gt;&gt;&gt;&gt;&gt;&gt; cdad75e521... <a href="https://jira.whamcloud.com/browse/LU-11069" title="ifort lseek returns wrong position on lustre 2.10.3" class="issue-link" data-issue-key="LU-11069"><del>LU-11069</del></a> llite: correct file position after appending writes</p> <p> If you agree, I will keep fields dv, rc and last_rc. The 'dv' field might have been removed in master branch in commits that 2.10.3 (b2_10) does not have.</p> <p>Yes, this is no problem. <tt>dv</tt> was moved to a different part of the file in another patch, but it is harmless to leave this as-is.</p> <p>Hi Jay,</p> <p>I encourage you to test the patch but please be aware that it has not completed full testing and review. In fact Jinshan pointed out an issue with the first patch set and I have submitted a second patch set. So we would not advise you to put this change into production just yet.</p> <p>More problems in the patch when cherry-picked to b2_10 without being flagged as conflicts. It then caused compilation errors.</p> <p>Could you please back port to b2_10? Thanks!</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32661" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32661</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11069" title="ifort lseek returns wrong position on lustre 2.10.3" class="issue-link" data-issue-key="LU-11069"><del>LU-11069</del></a> llite: correct file position after appending writes<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 1f2e3e935946f02950b67e9359a5d4d28216fbfd</p> <p>multiop.c: In function 'main': <br/> multiop.c:677:4: error: implicit declaration of function 'llapi_get_ost_layout_v<br/> ersion' <span class="error">&#91;-Werror=implicit-function-declaration&#93;</span> <br/>    rc = llapi_get_ost_layout_version(fd, &amp;layout_version); <br/>    ^ <br/> cc1: all warnings being treated as errors <br/> Makefile:1205: recipe for target 'multiop.o' failed <br/> make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;multiop.o&#93;</span> Error 1</p> <p>Hi Jay,</p> <p>A port of the change to b2_10 is available at <a href="https://review.whamcloud.com/#/c/32661/2" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32661/2</a>.</p> <p>The patch fixed the lseek issue in our fortran test. When can we expect for it to land.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ date &gt; tt $ date &gt;&gt; tt $ strace ./a.out &gt;&gt; tt .... ioctl(1, TCGETS, 0x7fffffff77a0)        = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) ioctl(1, TCGETS, 0x7fffffffbd00)        = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) fstat(1, {st_mode=S_IFREG|0600, st_size=58, ...}) = 0 ioctl(1, TCGETS, 0x7fffffffbd00)        = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) write(1, <span class="code-quote">" hello world\n"</span>, 13)          = 13 lseek(1, 0, SEEK_CUR)                   = 71 ftruncate(1, 71)                        = 0 ..... $ cat tt Fri Jun  8 16:00:58 PDT 2018 Fri Jun  8 16:01:01 PDT 2018  hello world </pre> </div></div> <p>Mahmoud</p> <p>If all goes well it should land to master mid-next week.</p> <p>Peter</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32641/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32641/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11069" title="ifort lseek returns wrong position on lustre 2.10.3" class="issue-link" data-issue-key="LU-11069"><del>LU-11069</del></a> llite: correct file position after appending writes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f8df907e8aaeaee559f913a68a345011a7aa1f23</p> <p>Landed for 2.12</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32661/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32661/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11069" title="ifort lseek returns wrong position on lustre 2.10.3" class="issue-link" data-issue-key="LU-11069"><del>LU-11069</del></a> llite: correct file position after appending writes<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 7bc18a0083b27292dd912dba0ae1aa5010568fb4</p> Duplicate Related LU-3044 LU-8964 Development Rank 1|hzzy0f: Rank (Obsolete) 9223372036854775807 Severity