Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-3026] Failure on test suite sanity-benchmark test_iozone https://jira.whamcloud.com/browse/LU-3026 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/898c8c02-94ba-11e2-93c6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/898c8c02-94ba-11e2-93c6-52540035b04c</a>.</p> <p>The sub-test test_iozone failed with the following error:</p> <blockquote> <p>iozone (1) failed</p></blockquote> <p>I will try to reproduce this issue manually to get more information</p> LU-3026

Failure on test suite sanity-benchmark test_iozone

Bug Blocker Resolved Fixed Jinshan Xiong Maloo HB Mon, 25 Mar 2013 17:39:51 +0000 Tue, 9 Apr 2013 00:10:55 +0000 Tue, 9 Apr 2013 00:10:55 +0000 Lustre 2.4.0 Lustre 2.4.0 0 7 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 3845408 512 Sanity check failed. Do not deploy this filesystem in a production environment ! </pre> </div></div> <p>It seems iozone detected some stability issue in the FS. 5.00% of last 100 executions</p> <p>Minh</p> <p>Could you please see if you are able to reproduce this and provide more data?</p> <p>Thanks</p> <p>Peter</p> <p>Also seen in DNE testing(1MDS/2MDTs)<br/> <a href="https://maloo.whamcloud.com/test_sets/872b7398-9658-11e2-9abb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/872b7398-9658-11e2-9abb-52540035b04c</a></p> <p>U-3060 After upgrade from 1.8.9 to 2.4, hit FAIL: iozone did not fail with EDQUOT</p> <p>Looks to be the same issue in a different context? </p> <p>Initially, this looks like <a href="https://jira.whamcloud.com/browse/LU-2909" title="Failure on test suite sanity-benchmark test_fsx: fsx bus error, core dumped" class="issue-link" data-issue-key="LU-2909"><del>LU-2909</del></a>. I am going to reproduce this and apply the patch to see if it fixes.</p> <p>I've done more runs and found that iozone only fail in auster run which use sanityusr to run. It passed when iozone run as root.</p> <p>A look at the iozone source found that ftruncate on a 0-byte file failed. I wrote a little program below to confirm this issue. If we run the executable as non-root, it will fail with "permission denied". This seem like a layout lock issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #define string_len 0 main() { <span class="code-object">int</span> file_descriptor; <span class="code-object">char</span> fn[]=<span class="code-quote">"write.file"</span>; struct stat st; <span class="code-keyword">if</span> ((file_descriptor = open(fn, (<span class="code-object">int</span>)O_CREAT|O_WRONLY,0)) < 0) perror(<span class="code-quote">"creat() error"</span>); <span class="code-keyword">else</span> { <span class="code-keyword">if</span> (ftruncate(file_descriptor, 0) != 0) perror(<span class="code-quote">"ftruncate() error"</span>); <span class="code-keyword">else</span> { fstat(file_descriptor, &st); printf(<span class="code-quote">"the file has %ld bytes\n"</span>, (<span class="code-object">long</span>) st.st_size); } close(file_descriptor); } } </pre> </div></div> <p>~ </p> <p>It turns out this is a problem of permission check on the MDT where __mdd_permission_internal() returns -EACCES because the file was created without access mode setting correctly.</p> <p>We can fix this problem by bypassing permission check on the MDT if it the permission check passed with file handler on the client side.</p> <p>I have tracked this down to <a href="https://build.whamcloud.com/job/lustre-master/1334/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/1334/</a> start breaking iozone</p> <p>patch is at here: <a href="http://review.whamcloud.com/5924" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5924</a>, this is imported by bb68e4c1.</p> <p>I verified that the patch worked on iozone</p> <p>> It turns out this is a problem of permission check on the MDT where __mdd_permission_internal() returns -EACCES because the file was created without access mode setting correctly.</p> <p>was created without access mode setting correctly? please clarify on this.</p> <blockquote> <p>was created without access mode setting correctly? please clarify on this.</p></blockquote> <p>Reproduce program is there, please try it.</p> <p>well, I was rather asking for a better explanation..</p> <p>ah I thought you're questioning how the file was created.</p> <p>For VFS, ftruncate does permission check by file's f_mode, since the file was created with O_WRONLY so it can pass w/o any problem. However, we do permission check on the MDT by inode mode which is not set at all.</p> <p>Ideally we should pass along file's mode but I choose to set MDS_OWNEROVERRIDE which seems easier. But I don't understand the code in mdd_fix_attr():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (la->la_valid & (LA_SIZE | LA_BLOCKS)) { <span class="code-keyword">if</span> (!((flags & MDS_OWNEROVERRIDE) && (uc->uc_fsuid == tmp_la->la_uid)) && !(flags & MDS_PERM_BYPASS)) { rc = mdd_permission_internal(env, obj, tmp_la, MAY_WRITE); <span class="code-keyword">if</span> (rc != 0) RETURN(rc); } } </pre> </div></div> <p>why it checks permission only if size and blocks are going to changed.</p> <p>are you saying iozone changes file's rights or own uid/gid once the file is open? if, so then the patch should be OK.</p> <p>given we do not pass openhandle to setattr, passing f_mode to MDS is not any better than MDS_OWNEROVERRIDE, but more code.</p> <p>the code pasted above seem to be solving this specific problem - to let process passed permission checks in open to be able to modify the file even if the actual rights have changed.</p> <p>would be good to mention the case in the code or in the patch.</p> <blockquote> <p>are you saying iozone changes file's rights or own uid/gid once the file is open? if, so then the patch should be OK.</p></blockquote> <p>No, there is no changing mode after file is opened. The file is created with O_WRONLY but mode is not set in open(2).</p> <blockquote> <p>the code pasted above seem to be solving this specific problem - to let process passed permission checks in open to be able to modify the file even if the actual rights have changed.</p></blockquote> <p>To be honest, I don't think it can pass permission check in anyways. However, changing rights after opening the file is a tricky case so I'm okay with the current code</p> <p>> No, there is no changing mode after file is opened. The file is created with O_WRONLY but mode is not set in open(2).</p> <p>hmm, then why current rights is not enough to proceed?</p> <p>The file is created w/o mode:</p> <p><span class="error">[root@jupiter ~]</span># ls -l /mnt/lustre/ttt/write.file <br/> ---------- 1 root root 0 Apr 6 09:06 /mnt/lustre/ttt/write.file</p> <p>ah, nice. thanks. then sure, MDS_OWNEROVERRIDE sounds correct.</p> <p>oh, wait.. but this should apply only to ftruncate()? how do we make sure regular truncate() won't bypass the checks?</p> <p>for truncate(), it won't pass in kernel check on the client side.</p> <p>Landed for 2.4</p> Development Rank 1|hzvm8v: Rank (Obsolete) 7389 Severity