https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3849 Lustre <p>Hi,</p> <p>On a customer cluster, we noticed that Lustre client cache directory information is not updated when directory owner or access mode changes.</p> <p>If a user tries to change owner/group or directory mode and then looks for the change from another node, the change is not visible. Moreover, the more we look, the more the cache update is delayed.</p> <p>We were suspecting troubles related to <a href="https://jira.whamcloud.com/browse/LU-3671" title="why are permission changes synchronous?" class="issue-link" data-issue-key="LU-3671"><del>LU-3671</del></a>, but we checked that sync_permission is set to 1.</p> <p>This problem is easy to reproduce:<br/> Node 1 : make directory<br/> with umask 0027<br/> mkdir &lt;lustre filesystem&gt;/&lt;new directory&gt;</p> <p> Node 2 : add directory in node2 cache<br/> cd &lt;lustre filesystem&gt;/&lt;new directory&gt;<br/> ls -lia</p> <p> Node 1 : update directory<br/> chmod 770 &lt;lustre filesystem&gt;/&lt;new directory&gt;<br/> or<br/> chown<br/> or both</p> <p> Node 2 : look for the change<br/> ls -lia (don't wait more than 10s)</p> <p>Thanks,<br/> Sebastien.</p> LU-3849

Client cache directory information is not updated

Bug Major Resolved Duplicate Lai Siyao Sebastien Buisson Wed, 28 Aug 2013 15:07:20 +0000 Wed, 16 Oct 2013 03:05:22 +0000 Fri, 11 Oct 2013 21:20:04 +0000 Lustre 2.1.5 0 9 <p>Lai</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>This issue should have existed for quite some time:</p> <ul> <li>open dir by `ls -lia` fetches CR LOOKUP|OPEN lock.</li> <li>setattr by `chown ...` enqueues a PW LOOKUP lock.</li> </ul> <p>But CR and PW mode are compatible, so the first lock is not canceled, and client get old stats.</p> <p>Changing setattr lock mode from PW to EX can fix this, I'll commit a fix later.</p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a></p> <p>Alex,</p> <p>You mean patch at <a href="http://review.whamcloud.com/6460" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6460</a> will fix this issue?</p> <p>i'm not a fully sure, but that patch disable matching a some locks's locally, a specially client will skip open lock if they need attributes for an inode (<a href="http://review.whamcloud.com/#/c/6460/3/lustre/mdc/mdc_locks.c" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6460/3/lustre/mdc/mdc_locks.c</a>).</p> <p>also readdir isn't create a lock to match to get an valid attributes, so i suggest to ask you to test that patch before Liang will create a new patch.</p> <p>I have reproduced this error on lustre 2.1.6 and lustre 2.4.0.</p> <p>I have backported this patch on lustre 2.4 and I will test it next week.</p> <p>I have made some additional tests, and I have noticed that the information printed by "ls" is correct if we specify the directory.<br/> "ls -la ." or "ls -la $PWD" on "node 2" prints the current (correct) inode information.</p> <p>But if we issue a chmod/chown/chgrp/mkdir/rmdir on "node 1", and then a "ls" on "node 2", without specifying a "directory name", and whatever option is specified, then the inode information printed does not reflect the last change made on "node 1". Moreover, if we issue a second "ls" command, but this time adding "." or "$PWD", the printed inode information is still bad.</p> <p>Looking at "ls" source code, it appears that the code starts with an "opendir()" and a loop of "readdir()" on the current directory, when no file or directory is specified. If a name is specified, then the command starts by "stat()" on this name.</p> <p>I wrote the following simple command, allowing to show that the culprit is readdir().</p> <p>The command first issues an opendir() of "." and a loop of readdir()/printf() , but only if an argument is specified.<br/> It then issues a stat() on ".", and prints some inode fields.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#include &lt;sys/types.h&gt; #include &lt;errno.h&gt; #include &lt;sys/stat.h&gt; #include &lt;dirent.h&gt; #include &lt;stdio.h&gt; #include &lt;unistd.h&gt; #include &lt;stdlib.h&gt; main(int argc, char **argv) { struct stat dir_stat; DIR *dirp; struct dirent *next; if (argc &gt; 1) { dirp = opendir("."); if (! dirp) { perror("opendir: "); exit(1); } while (next = readdir(dirp)) { printf("%s ", next-&gt;d_name); } printf("\n"); closedir(dirp); } if (stat(".", &amp;dir_stat) &lt; 0) { perror("stat: "); exit(1); } printf("st_nlink %d, st_mode 0%ho, st_uid 0x%hx, st_gid 0x%hx\n", dir_stat.st_nlink, dir_stat.st_mode, dir_stat.st_uid, dir_stat.st_gid); exit(0); } </pre> </div></div> <p>To reproduce the error, you have to run the chmod/chown/chgrp/mkdir/rmdir commands on "node 1", and after each one, run "my_stat --" on "node 2".<br/> The changes made on "node 1" are not reflected on "node 2".</p> <p>Now re-issue one of the commands on "node 1", and run "my_stat" without argument on "node 2". All the changes made previously are now visible.</p> <p>I have also checked that the problem described in <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a> is indentical, but only with mkdir/rmdir when only one node is in use.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>for i in `seq 1 10`; do mkdir d$i my_stat -- done mkdir last my_stat </pre> </div></div> <p>During the loop, the st_nlink is not correct (always 2, instead of 3, 4, 5, ...).<br/> At the end of the loop, after issueing a new mkdir, and then a stat() without readdir(), the st_nlink=13 is the correct value.</p> <p>In my test I found another issue:</p> <ul> <li>ll_get_dir_page() may enqueue UPDATE lock, but it doesn't inode attributes, so these informations may look stale on client side with UPDATE lock held.</li> </ul> <p>Lyang,</p> <p>problem is - ll_get_dir_page take a lock but don't take a intent body so don't update inode with info from server.</p> <p>&gt; During the loop, the st_nlink is not correct (always 2, instead of 3, 4, 5, ...).<br/> take a patch, it's looks same problem and test looks same <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> please look to sanity 39o test.</p> <p>Alexey, Lai,<br/> I tested set 4 of patch <a href="http://review.whamcloud.com/6460" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6460</a> on a lustre 2.4.0 client, and it fixes the issue described above, as well as the issue described in <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a>.</p> <p>Thanks to testing! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I hope Intel will found time to land it patch.</p> <p>Alexey, the patch in question has a -1 code review which will need to be resolved (additional explanation, code fix, whatever is needed) before before landing can be expected.</p> <p>Christopher,</p> <p>I think we need to choose a different reviewer because questions is very strange and can't be implemented without MDT rewrite (a specially mdt_getattr_name_lock don't able to return lock without look to parent).<br/> If IT_READIR was don't drop on 2.x protocol it's more easy to fix, but now it's need too much compatibility code.</p> <p>Alexey,</p> <p>Can you respond to Fan Young's comments in gerrit and maybe we can find a way to move forward with the patch?</p> <p>This problem is very similar to <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a> and will be fixed by the patch on that bug.</p> Related LU-3240 Development Rank 1|hzvzi7: Rank (Obsolete) 9968 Severity