https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6813 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/37318204-250a-11e5-8009-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/37318204-250a-11e5-8009-5254006e85c2</a>.</p> <p>The sub-test test_iozone failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>iozone (1) failed </pre> </div></div> <p>test log. Hit this problem in multiple configs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 3847216 512 Sanity check failed. Do not deploy this filesystem in a production environment ! sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed </pre> </div></div> <p>MDS console, it looks like there is not enough space</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:15:50:Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone ============================================================== 11:04:05 (1436205845) 18:15:51:Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 1846488kB available, using 3847216kB file size 18:15:51:Lustre: DEBUG MARKER: min OST has 1846488kB available, using 3847216kB file size 18:15:51:Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-benchmark test_iozone: @@@@@@ FAIL: iozone \(1\) failed 18:15:51:Lustre: DEBUG MARKER: sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed 18:15:51:Lustre: DEBUG MARKER: /usr/sbin/lctl dk &gt; /logdir/test_logs/2015-07-06/lustre-master-el6_6-x86_64-vs-lustre-master-sles11sp3-x86_64--full--2_12_1__3092__-70321095799900-131112/sanity-benchmark.test_iozone.debug_log.$(hostname -s).1436205847.log; </pre> </div></div> LU-6813

sanity-benchmark test_iozone: iozone (1) failed

Bug Critical Resolved Fixed Zhenyu Xu Maloo Wed, 8 Jul 2015 17:10:03 +0000 Thu, 12 May 2016 18:12:41 +0000 Wed, 23 Sep 2015 13:07:09 +0000 Lustre 2.8.0 Lustre 2.8.0 0 9 <p>I don't see any reports on the MDS or OSS logs that relate to out of space. It looks like this might be a data corruption issue, since there aren't errors in any of the logs. The message <tt>min OST has 1846488kB available, using 3847216kB file size</tt> is just reporting how large the iozone file is. The file is striped across all OSTs, so it shouldn't consume all of the space on even the smallest OST, since each OST will get only 1/7 of the file data (549602KB, 29% of the free space).</p> <p>Hello Andreas,</p> <p>We have also faced this issue much frequently in our testing. I am attaching the logs to this ticket. Please let me know if you need any extra information from our side. </p> <p>found this with strace </p> <p>open("/mnt/lustre/d0.iozone/iozone", O_WRONLY|O_CREAT, 0) = 3<br/> stat("/mnt/lustre/d0.iozone/iozone", </p> {st_mode=S_IFREG, st_size=0, ...} <p>) = 0<br/> ftruncate(3, 0) = -1 EACCES (Permission denied)<br/> write(1, "\n\nSanity check failed. Do not de"..., 82</p> <p>the iozone test file is created by open with mode 0100000(S_IFREG), no WRX mode in it.</p> <p>00000004:00000002:1.0:1442290645.210840:0:15269:0:(mdt_open.c:1226:mdt_reint_open()) I am going to open <span class="error">&#91;0x200000401:0x11:0x0&#93;</span>/(iozone-&gt;<span class="error">&#91;0x200000401:0x15:0x0&#93;</span>) cr_flag=0102 mode=0100000 msg_flag=0x0</p> <p>commit c8d5aa14e50be2a85491783f169a8f4e646b9594 changed the object create mode logic.</p> <p>Wang Di, can you give it a look? In your commit, the MDS does not use its umask to create object and client does not pass the object's mode in the RPC, and client is depending on MDS to set the new object's mode with its umask.</p> <p>client create the file with strange mode</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:1.0:1442292147.237309:0:28484:0:(namei.c:526:ll_lookup_it()) Process entered 00000080:00200000:1.0:1442292147.237309:0:28484:0:(namei.c:533:ll_lookup_it()) VFS Op:name=iozone, dir=[0x200000401:0x1:0x0](ffff880016c0bc40), intent=open|creat 00000080:00000010:1.0:1442292147.237311:0:28484:0:(llite_lib.c:2436:ll_prep_md_op_data()) kmalloced 'op_data': 312 at ffff8800286e1a00. 00000080:00020000:1.0:1442292147.237312:0:28484:0:(namei.c:561:ll_lookup_it()) create mode 0100000 </pre> </div></div> <p>while normal touch create a file with normal mode</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:0.0:1442298104.737478:0:28661:0:(namei.c:526:ll_lookup_it()) Process entered 00000080:00200000:0.0:1442298104.737478:0:28661:0:(namei.c:533:ll_lookup_it()) VFS Op:name=touchfile, dir=[0x200000401:0x1:0x0](ffff880016c0bc40), intent=open|creat 00000080:00000010:0.0:1442298104.737480:0:28661:0:(llite_lib.c:2436:ll_prep_md_op_data()) kmalloced 'op_data': 312 at ffff8800352f2400. 00000080:00020000:0.0:1442298104.737482:0:28661:0:(namei.c:561:ll_lookup_it()) create mode 0100666 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Wang Di, can you give it a look? In your commit, the MDS does not use its umask to create object and client does not pass the object's mode in the RPC, </pre> </div></div> <p>Hmm, mdd_acl_init do use MDS's umask to fix the mode. Thanks.</p> <p>the strace iozone shows that iozone create the testing file "iozone" without setting its file mode</p> <p>open("/mnt/lustre/d0.iozone/iozone", O_WRONLY|O_CREAT, 0) = 3</p> <p>And client pack its create mode as 0100000, no file mode in it.</p> <p>In mdd_acl_init(), la-&gt;la_mode is still just 0100000, and the file is created with empty mode, so that truncate failed permission checking.</p> <p>$ git bisect bad<br/> 6acf93339ad3297f2e5c659f2269c05df6198f74 is the first bad commit<br/> commit 6acf93339ad3297f2e5c659f2269c05df6198f74<br/> Author: Jinshan Xiong &lt;jinshan.xiong@intel.com&gt;<br/> Date: Sun Jun 21 14:29:20 2015 -0400</p> <p> <a href="https://jira.whamcloud.com/browse/LU-5823" title="Replace some obsolete obd operations with CLIO ioctl interface" class="issue-link" data-issue-key="LU-5823"><del>LU-5823</del></a> llite: Remove access of stripe in ll_setattr_raw</p> <p>wrote a open then ftruncate test code, run it as a normal user, gathered the logs w/ and w/o the 6acf9333 patch.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16462" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16462</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6813" title="sanity-benchmark test_iozone: iozone (1) failed" class="issue-link" data-issue-key="LU-6813"><del>LU-6813</del></a> llite: omit to update wire data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6d32529f5c853f8a2e800fc588dc396be9b07405</p> <p>We ran the test_iozone on the patch submitted by Bobi Jam and the test was successful. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 926070 512 40674 66809 161970 151712 164405 61944 iozone test complete. debug=0x33f0484 Resetting fail_loc on all nodes...done. PASS iozone (103s) </pre> </div></div> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16462/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16462/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6813" title="sanity-benchmark test_iozone: iozone (1) failed" class="issue-link" data-issue-key="LU-6813"><del>LU-6813</del></a> llite: omit to update wire data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b937be892a3dc68dd2fe3f248608937a8a79d424</p> <p>Landed for 2.8</p> Related Development Rank 1|hzxhpr: Rank (Obsolete) 9223372036854775807 Severity