https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10335 Lustre <p>Here is the Maloo link <a href="https://testing.hpdd.intel.com/test_sets/cad15292-db78-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cad15292-db78-11e7-9c63-52540065bddc</a></p> <p>test_130a console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 130a: FIEMAP (1-stripe file) ========================================================== 01:03:11 (1512522191) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00117857 s, 55.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f130a.sanity is 65536 (16 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: /usr/lib64/lustre/tests/sanity.sh: line 9256: 25375 Aborted (core dumped) filefrag -ves $fm_file sanity test_130a: @@@@@@ FAIL: filefrag /mnt/lustre/f130a.sanity failed </pre> </div></div> <p>test_130b/c/e</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 01:03:25 (1512522205) /mnt/lustre/f130e.sanity: FIBMAP unsupported Filesystem type is: bd00bd0 File size of /mnt/lustre/f130e.sanity is 67043328 (16368 blocks of 4096 bytes) </pre> </div></div> server: 2.10.2 RC1<br/> client: Ubuntu16.04 LU-10335

Ubuntu1604 client sanity-130a: FAIL: filefrag -ves core dumped

Bug Major Resolved Won't Do Sarah Liu Sarah Liu ubuntu Wed, 6 Dec 2017 05:41:24 +0000 Wed, 27 May 2020 02:46:03 +0000 Fri, 10 Aug 2018 03:42:50 +0000 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.10.4 Lustre 2.12.0 0 5 <p>This looks like the error message reported in <a href="https://jira.whamcloud.com/browse/LU-10333" title="Ubuntu1604 client sanity-255a hung: buffer overflow detected " class="issue-link" data-issue-key="LU-10333"><del>LU-10333</del></a>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>*** buffer overflow detected ***: filefrag terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff72cd1c7e5] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7ff72cdbe11c] /lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7ff72cdbc120] /lib/x86_64-linux-gnu/libc.so.6(+0x116689)[0x7ff72cdbb689] /lib/x86_64-linux-gnu/libc.so.6(_IO_default_xsputn+0x80)[0x7ff72cd206b0] /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0xc90)[0x7ff72ccf2e00] /lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x84)[0x7ff72cdbb714] /lib/x86_64-linux-gnu/libc.so.6(__sprintf_chk+0x7d)[0x7ff72cdbb66d] filefrag[0x4018c8] filefrag[0x401ccf] filefrag[0x4012f5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff72ccc5830] filefrag[0x4015c9] ======= Memory map: ======== </pre> </div></div> <p>What version of e2fsprogs is installed on the Ubuntu client?</p> <p>Sarah, could you please report what version of <tt>filefrag</tt> is installed on the Ubuntu client, <tt>rpm -qf $(which filefrag)</tt>. I suspect it is the unpatched filefrag, which will not be able to run test_130b/c/e, but it should be able to run test_130a without crashing.</p> <p>Could you (or someone) please login to an Ubuntu client and run <tt>filefrag</tt> under gdb to get a stack trace? You may need to install the <tt>e2fsprogs-debug</tt> RPM to get useful information from the crash.</p> <p>Hi Andreas,</p> <p>just checked and there is no filefrag installed on the Ubuntu client</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@onyx-21vm3:~# dpkg -l|grep -i filefra root@onyx-21vm3:~# dpkg -l|grep -i e2fsprogs ii e2fsprogs 1.42.13-1ubuntu1 amd64 ext2/ext3/ext4 file system utilities </pre> </div></div> <p>The filefrag program is part of e2fsprogs.</p> <p>Sarah, could you please run filefrag under gdb and collect the stack trace. </p> <p>Ok, I will update the ticket when I get the trace</p> <p>this is what I got:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) set args -ves /mnt/lustre/file-130a (gdb) run Starting program: /usr/sbin/filefrag -ves /mnt/lustre/file-130a Filesystem type is: bd00bd0 File size of /mnt/lustre/file-130a is 65536 (16 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: *** buffer overflow detected ***: /usr/sbin/filefrag terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff7a847e5] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7ffff7b2615c] /lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7ffff7b24160] /lib/x86_64-linux-gnu/libc.so.6(+0x1166c9)[0x7ffff7b236c9] /lib/x86_64-linux-gnu/libc.so.6(_IO_default_xsputn+0x80)[0x7ffff7a886b0] /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0xc90)[0x7ffff7a5ae00] /lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x84)[0x7ffff7b23754] /lib/x86_64-linux-gnu/libc.so.6(__sprintf_chk+0x7d)[0x7ffff7b236ad] /usr/sbin/filefrag[0x4018c8] /usr/sbin/filefrag[0x401ccf] /usr/sbin/filefrag[0x4012f5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff7a2d830] /usr/sbin/filefrag[0x4015c9] ======= Memory map: ======== 00400000-00403000 r-xp 00000000 fd:01 2231256 /usr/sbin/filefrag 00602000-00603000 r--p 00002000 fd:01 2231256 /usr/sbin/filefrag 00603000-00604000 rw-p 00003000 fd:01 2231256 /usr/sbin/filefrag 00604000-00626000 rw-p 00000000 00:00 0 [heap] 7ffff77f7000-7ffff780d000 r-xp 00000000 fd:01 262665 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff780d000-7ffff7a0c000 ---p 00016000 fd:01 262665 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff7a0c000-7ffff7a0d000 rw-p 00015000 fd:01 262665 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff7a0d000-7ffff7bcd000 r-xp 00000000 fd:01 266956 /lib/x86_64-linux-gnu/libc-2.23.so 7ffff7bcd000-7ffff7dcd000 ---p 001c0000 fd:01 266956 /lib/x86_64-linux-gnu/libc-2.23.so 7ffff7dcd000-7ffff7dd1000 r--p 001c0000 fd:01 266956 /lib/x86_64-linux-gnu/libc-2.23.so 7ffff7dd1000-7ffff7dd3000 rw-p 001c4000 fd:01 266956 /lib/x86_64-linux-gnu/libc-2.23.so 7ffff7dd3000-7ffff7dd7000 rw-p 00000000 00:00 0 7ffff7dd7000-7ffff7dfd000 r-xp 00000000 fd:01 266954 /lib/x86_64-linux-gnu/ld-2.23.so 7ffff7fea000-7ffff7fed000 rw-p 00000000 00:00 0 7ffff7ff6000-7ffff7ff7000 rw-p 00000000 00:00 0 7ffff7ff7000-7ffff7ffa000 r--p 00000000 00:00 0 [vvar] 7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso] 7ffff7ffc000-7ffff7ffd000 r--p 00025000 fd:01 266954 /lib/x86_64-linux-gnu/ld-2.23.so 7ffff7ffd000-7ffff7ffe000 rw-p 00026000 fd:01 266954 /lib/x86_64-linux-gnu/ld-2.23.so 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Program received signal SIGABRT, Aborted. 0x00007ffff7a42428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) backtrace #0 0x00007ffff7a42428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007ffff7a4402a in __GI_abort () at abort.c:89 #2 0x00007ffff7a847ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff7b9c49f "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007ffff7b2615c in __GI___fortify_fail (msg=&lt;optimized out&gt;, msg@entry=0x7ffff7b9c430 "buffer overflow detected") at fortify_fail.c:37 #4 0x00007ffff7b24160 in __GI___chk_fail () at chk_fail.c:28 #5 0x00007ffff7b236c9 in _IO_str_chk_overflow (fp=&lt;optimized out&gt;, c=&lt;optimized out&gt;) at vsprintf_chk.c:31 #6 0x00007ffff7a886b0 in __GI__IO_default_xsputn (f=0x7fffffffa580, data=&lt;optimized out&gt;, n=8) at genops.c:455 #7 0x00007ffff7a5ae00 in _IO_vfprintf_internal (s=s@entry=0x7fffffffa580, format=&lt;optimized out&gt;, format@entry=0x402124 "%#04x,", ap=ap@entry=0x7fffffffa6b8) at vfprintf.c:1631 #8 0x00007ffff7b23754 in ___vsprintf_chk (s=0x7fffffffa7c0 "0x800", flags=1, slen=6, format=0x402124 "%#04x,", args=args@entry=0x7fffffffa6b8) at vsprintf_chk.c:82 #9 0x00007ffff7b236ad in ___sprintf_chk (s=&lt;optimized out&gt;, flags=&lt;optimized out&gt;, slen=&lt;optimized out&gt;, format=&lt;optimized out&gt;) at sprintf_chk.c:31 #10 0x00000000004018c8 in ?? () ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- #11 0x0000000000401ccf in ?? () #12 0x00000000004012f5 in ?? () #13 0x00007ffff7a2d830 in __libc_start_main (main=0x400b00, argc=3, argv=0x7fffffffec08, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, rtld_fini=&lt;optimized out&gt;, stack_end=0x7fffffffebf8) at ../csu/libc-start.c:291 #14 0x00000000004015c9 in ?? () (gdb) </pre> </div></div> <p>I rebuild e2fsprogs with debug symbols but cannot hit the problem with the updated filefrag</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@onyx-24vm1:~# file /usr/sbin/filefrag /usr/sbin/filefrag: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=6c421f2064cfcb7aff7314dcb9db4f380e7378f0, not stripped root@onyx-24vm1:~# gdb filefrag GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: &lt;http://www.gnu.org/software/gdb/bugs/&gt;. Find the GDB manual and other documentation resources online at: &lt;http://www.gnu.org/software/gdb/documentation/&gt;. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from filefrag...done. (gdb) set args -ves /mnt/lustre/foo (gdb) run Starting program: /usr/sbin/filefrag -ves /mnt/lustre/foo Filesystem type is: bd00bd0 File size of /mnt/lustre/foo is 65536 (16 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 15: 35346.. 35361: 16: last,0x80000000,eof /mnt/lustre/foo: 1 extent found [Inferior 1 (process 10604) exited normally] (gdb) quit </pre> </div></div> <p>The problem was that the unpatched Ubuntu e2fsprogs was printing unknown flags into a temporary buffer, but Lustre always sets the <tt>0x80000000</tt> flag to identify network-based filesystems. However, this overflows the temporary buffer, which was only 6 bytes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* print any unknown flags as hex values */</span> <span class="code-keyword">for</span> (mask = 1; fe_flags != 0 &amp;&amp; mask != 0; mask &lt;&lt;= 1) { <span class="code-object">char</span> hex[6]; <span class="code-keyword">if</span> ((fe_flags &amp; mask) == 0) <span class="code-keyword">continue</span>; sprintf(hex, <span class="code-quote">"%#04x,"</span>, mask); print_flag(&amp;fe_flags, mask, flags, hex); } </pre> </div></div> <p>Any unknown flag would overflow this, since it would always have <b>at least</b> 4 hex digits, plus the leading <tt>0x</tt> and a trailing <tt>NUL</tt>, so at least 7 characters printed each time. I've submitted a patch upstream for this.</p> <p>According to <a href="https://marc.info/?l=linux-ext4&amp;m=152010285623799&amp;w=2" class="external-link" target="_blank" rel="nofollow noopener">https://marc.info/?l=linux-ext4&amp;m=152010285623799&amp;w=2</a> the patch was landed, but was subsequently lost from the tree:</p> <blockquote> <p>List: linux-ext4<br/> Subject: Re: <span class="error">&#91;PATCH&#93;</span> filefrag: avoid temporary buffer overflow<br/> From: Theodore Ts'o &lt;tytso () mit ! edu&gt;<br/> Date: 2018-03-03 18:47:12<br/> Message-ID: 20180303184712.GA26224 () thunk ! org</p> <p>On Fri, Mar 02, 2018 at 09:48:28AM -0800, Darrick J. Wong wrote:<br/> &gt; On Thu, Mar 01, 2018 at 01:09:46PM -0700, Andreas Dilger wrote:<br/> &gt; &gt; From: Andreas Dilger &lt;adilger@dilger.ca&gt;<br/> &gt; &gt; <br/> &gt; &gt; If an unknown flag is present in a FIEMAP extent, it is printed as a<br/> &gt; &gt; hex value into a temporary buffer before adding it to the flags. If<br/> &gt; &gt; that unknown flag is over 0xffff then it will overflow the temporary<br/> &gt; &gt; buffer.<br/> &gt; &gt; <br/> &gt; &gt; Reported-by: Sarah Liu &lt;wei3.liu@intel.com&gt;<br/> &gt; &gt; Intel-bug-id: <a href="https://jira.hpdd.intel.com/browse/LU-10335" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-10335</a><br/> &gt; &gt; Signed-off-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt;<br/> &gt; <br/> &gt; Looks ok,<br/> &gt; Reviewed-by: Darrick J. Wong &lt;darrick.wong@oracle.com&gt;</p> <p>Thanks, applied with the 0xfff fixup in the commit description.</p> <ul class="alternate" type="square"> <li>Ted</li> </ul> </blockquote> <p>I've resubmitted the patch, and asked that it also be landed to the Debian maintenance branch so it will appear in Ubuntu.</p> <p>This has landed to upstream e2fsprogs for the 1.45 and 1.44.2 releases:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 17a1f2c1929630e3a79e6b98168d56f96acf2e8b Author: Andreas Dilger &lt;adilger@dilger.ca&gt; AuthorDate: Thu Mar 29 12:36:54 2018 -0600 Commit: Theodore Ts'o &lt;tytso@mit.edu&gt; CommitDate: Thu Mar 29 23:01:19 2018 -0400 filefrag: avoid temporary buffer overflow If an unknown flag is present in a FIEMAP extent, it is printed as a hex value into a temporary buffer before adding it to the flags. If that unknown flag is over 0xfff then it will overflow the temporary buffer. Reported-by: Sarah Liu &lt;wei3.liu@intel.com&gt; Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-10335 Signed-off-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt; Signed-off-by: Theodore Ts'o &lt;tytso@mit.edu&gt; </pre> </div></div> <p>Add to <tt>ALWAYS_EXCEPT</tt> for Ubuntu until they get an updated e2fsprogs release with this fix and/or we install our patched e2fsprogs.</p> <p>If the above patch has been included into the Ubuntu e2fsprogs (at least for the versions we are testing), then this ALWAYS_EXCEPT can be removed. If not, can you please file a ticket with the upstream Ubuntu bug tracker to have them backport the above patch from e2fsprogs master into their release.</p> <p>I just did a apt-get source e2fsprogs and checked for the fix you posted here. For Ubuntu18 the fix is there but its lacking in Ubuntu16. Since for 2.12 we have Ubuntu server support we will need the lustre special e2fsprogs anyways.</p> <p>The patched e2fsprogs does not have this problem, only the unpatched e2fsprogs. It is good that this is included in Ubuntu 18, and it isn't clear we can do anything about Ubuntu 16 at this point.</p> <p>Once we have patched e2fsprogs for ldiskfs support for Ubuntu this should go away.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33459" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33459</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10335" title="Ubuntu1604 client sanity-130a: FAIL: filefrag -ves core dumped" class="issue-link" data-issue-key="LU-10335"><del>LU-10335</del></a> test: enable sanity 130 tests for Ubuntu<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 59bab702aed6d10ae70336b8372a13e6169093ee</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33459/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33459/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10335" title="Ubuntu1604 client sanity-130a: FAIL: filefrag -ves core dumped" class="issue-link" data-issue-key="LU-10335"><del>LU-10335</del></a> test: enable sanity 130 tests for Ubuntu<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 41a099f9c03c2a3ff62360433985ea5de3e52962</p> Related LU-6007 LU-10997 LU-13177 Development Rank 1|hzzotz: Rank (Obsolete) 9223372036854775807 Severity