https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-615 Lustre <p>sanity test 180a failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 180a: test obdecho on osc == 02:30:13 (1314005413) awk: (FILENAME=/proc/fs/lustre/osc/lustre-OST0000-osc-ffff88032273d800/import FNR=32) fatal error: internal error </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/cdf213b2-ccac-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cdf213b2-ccac-11e0-8d02-52540025f9af</a></p> <br/> Lustre Tag: v2_1_0_0_RC0<br/> Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/267/">http://newbuild.whamcloud.com/job/lustre-master/267/</a><br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6)<br/> LU-615

sanity test 180a: awk: (FILENAME=/proc/fs/lustre/osc/lustre-OST0000-osc-ffff88032273d800/import FNR=32) fatal error: internal error

Bug Blocker Resolved Duplicate WC Triage Jian Yu Mon, 22 Aug 2011 07:47:26 +0000 Mon, 15 Jul 2013 19:57:05 +0000 Mon, 6 Feb 2012 13:31:54 +0000 Lustre 2.1.0 Lustre 2.2.0 0 3 <p>Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/275/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-master/275/</a><br/> Distro/Arch: RHEL5/x86_64(server, OFED 1.5.3.1), SLES11/x86_64(client, in-kernel OFED)</p> <p>sanity test 180a failed with the same issue: <a href="https://maloo.whamcloud.com/test_sets/e4d30bfa-d561-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e4d30bfa-d561-11e0-8d02-52540025f9af</a></p> <p>Probably related to <a href="https://jira.whamcloud.com/browse/LU-566" title="sanity test 180a: *** glibc detected *** awk: malloc(): memory corruption: 0x0000000000cb31b0 ***" class="issue-link" data-issue-key="LU-566"><del>LU-566</del></a>.</p> <p>we are seeing that bug very often in Orion over rhel6. I have an idea that awk has troubles parsing proc files. The patch copy proc file to the regular one and use it, there is fix:</p> <p><a href="http://review.whamcloud.com/1467" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1467</a></p> <p>the Maloo run with it shows test 180a pass, but this is only single run.</p> <p>It seems this is related to our implementation of proc read method there. The amount of data in 'import' is quite big, more than 1024 bytes. Strace shows the following while reading proc file:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">open(<span class="code-quote">"/proc/fs/lustre/osc/lustre-OST0001-osc-d9f1ea00/<span class="code-keyword">import</span>"</span>, O_RDONLY|O_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfdb9348) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 read(3, <span class="code-quote">"<span class="code-keyword">import</span>:\n name: lustre-OST0001"</span>..., 1024) = 1013 </pre> </div></div> <p>1013 bytes is on my local system, on Toro it is more than 1024, e.g. my local address is just 0@lo while on Toro it is full IP address.<br/> The issue here is fstat64 - it can't get file size for proc file, so following read is using 1024 for read which is less than proc data, probably that is the reason of issue. The lprocfs_rd_import() uses snprintf to output data but it will truncate data so last line may have no final end-of-line symbol, can this cause awk to fail?</p> <p>If we will use just tmp file with output of proc data that works fine because awk gets file size by fstat64, so this solution will work.<br/> But better to fix our proc read method for 'import' data, using seq_file.</p> <p>Redo 'import' proc file via seq_file API:</p> <p><a href="http://review.whamcloud.com/#change,1544" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1544</a></p> <p>I've made local tests to make sure it is able to read big amount of data, to be sure I've added several line of just garbage so output become bigger than 1024 and run awk with strace:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">open(<span class="code-quote">"/proc/fs/lustre/osc/lustre-OST0001-osc-d1a26e00/<span class="code-keyword">import</span>"</span>, O_RDONLY|O_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfe6bec8) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 read(3, <span class="code-quote">"<span class="code-keyword">import</span>:\n name: lustre-OST0001"</span>..., 1024) = 1024 read(3, <span class="code-quote">"hat is why I put <span class="code-keyword">this</span> line to ex"</span>..., 1024) = 341 fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb765f000 write(1, <span class="code-quote">"747\n"</span>, 4747 ) = 4 read(3, "", 1024) = 0 close(3) = 0 exit_group(0) = ? </pre> </div></div> <p>the command was:<br/> strace awk '/usec_per_rpc:/ </p> {print $2} <p>' /proc/fs/lustre/osc/lustre-OST0001-osc-d1a26e00/import</p> <p>it searched usec_per_rpc string which is at end of file and we can see that awk did two reads - 1024 bytes and 341 bytes, string was found in second part. </p> <p>Previous implementation lprocfs_rd_import() wasn't able to return data from offset at all and just cut data at 1024 causing awk to behave unpredictable </p> <p>hit the same issue when testing interop between 2.1-wc1 client and 2.1.55 server</p> <p>Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc == 19:30:56 (1328067056)<br/> Lustre: Echo OBD driver; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a></p> <ul> <li> <ul> <li> <ul> <li>glibc detected *** awk: malloc(): memory corruption: 0x0000000000f3a1b0 ***</li> </ul> </li> </ul> </li> </ul> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-760" title="Test failure on test suite sanity, subtest test_180a" class="issue-link" data-issue-key="LU-760"><del>LU-760</del></a></p> Related LU-760 Development Rank 1|hzvbsn: Rank (Obsolete) 5475 Severity