https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6087 Lustre <p>This issue was created by maloo for John Hammond &lt;john.hammond@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/25cf25bc-9661-11e4-854d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/25cf25bc-9661-11e4-854d-5254006e85c2</a>.</p> <p>The sub-test test_300b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mtime not change after create </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity 300b</p> LU-6087

sanity test_300b: mtime not change after create

Bug Major Resolved Fixed Niu Yawei Maloo Wed, 7 Jan 2015 16:15:44 +0000 Tue, 16 Jun 2015 23:44:01 +0000 Mon, 26 Jan 2015 23:01:07 +0000 Lustre 2.7.0 Lustre 2.7.0 0 10 <p>Another from an unrelated change <a href="https://testing.hpdd.intel.com/test_sets/fed31d82-9666-11e4-a74a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fed31d82-9666-11e4-a74a-5254006e85c2</a>.</p> <p>I wonder if this is due to clock skew amond the MDTs. The sleeps in 300b are only 1 second and I see upto 0.5 second drift among the VMs from one of the sessions:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@shadow-23 ~]# time pdsh -w shadow-23vm[8-12] date '+%s.%N' shadow-23vm9: 1420651079.679483075 shadow-23vm12: 1420651079.956219142 shadow-23vm10: 1420651079.600486555 shadow-23vm8: 1420651079.417814394 shadow-23vm11: 1420651079.995738528 real 0m0.111s user 0m0.001s sys 0m0.009s </pre> </div></div> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13272" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13272</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6087" title="sanity test_300b: mtime not change after create" class="issue-link" data-issue-key="LU-6087"><del>LU-6087</del></a> test: increase sleep times in sanity 300b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 69e272136b94bdbd3dfac805b24ac61a69e077f7</p> <p>The timestamps for operations are (or should be) controlled by the client. That is to ensure the client's updates are self-consistent (e.g. single-client make) and is no worse than if the clients are running NTP. </p> <p>On 4 local VMs I had about 0.5 second drift between the client and mdt0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ time pdsh -l root -Rssh -w t[0-3] date '+%s.%N' t0: 1420658127.115688630 # mdt0 t1: 1420658127.471434603 # mdt1-3 t2: 1420658127.660439409 # ost0-7 t3: 1420658126.647549504 # client real 0m0.133s user 0m0.048s sys 0m0.024s </pre> </div></div> <p>With 1 second sleeps I saw 6 failures in 10 runs of 300b.</p> <p>With 2 second sleeps I saw no failures in 10 runs of 300b.</p> <p>Then I synced my VMs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ pdsh -l root -Rssh -w t[0-3] ntpdate ntp1.sbcglobal.net t3: 7 Jan 13:16:48 ntpdate[25263]: step time server 68.94.156.17 offset 15.446993 sec t2: 7 Jan 13:16:48 ntpdate[15588]: step time server 68.94.156.17 offset 14.437126 sec t0: 7 Jan 13:16:48 ntpdate[10770]: step time server 68.94.156.17 offset 14.969598 sec t1: 7 Jan 13:16:48 ntpdate[12428]: step time server 68.94.156.17 offset 14.613771 sec $ time pdsh -l root -Rssh -w t[0-3] date '+%s.%N' t0: 1420658219.028891881 t1: 1420658219.031173939 t3: 1420658219.031840292 t2: 1420658219.036044339 real 0m0.135s user 0m0.048s sys 0m0.028s </pre> </div></div> <p>With 1 second sleeps I saw no failures in 10 runs of 300b. I also see no failures when running everything on a single node.</p> <p>I think this is just hiding an underlying bug. With a single client, all the timestamps for files should come from the client and not the server. That syncing the clocks between the client and server VMs fixes the problem only points more toward the fact that a server timestamp is being used somewhere.</p> <p>It would be interesting to change the clock on the MDS and OSS to be wildly out of sync with the client and each other, and then run sanity and sanityN (it is still running on a single node, with two mountpoints) to see what fails. That would make it much more obvious where the bad timestamps are coming from. </p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13396" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13396</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6087" title="sanity test_300b: mtime not change after create" class="issue-link" data-issue-key="LU-6087"><del>LU-6087</del></a> mdd: update timestamps arbitrarily<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 216d148039da83c9af16c76b15f86ad940cf29e0</p> <p>Perhaps the stripe dir code used server time mistakenly somehow? I didn't see from the code yet. Di, could you take a look when you have time?</p> <p>Current timestamps update policy doesn't make sense to me, I think we'd fix it anyway: <a href="http://review.whamcloud.com/#/c/13396/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13396/</a></p> <p>I checked the code, and it seems the ctime/mtime got from client has been set on every stripe correctly.</p> <p>In my local testing <a href="http://review.whamcloud.com/#/c/13396/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13396/</a> did not fix this issue. I still saw about 20% failure.</p> <p>&gt; It would be interesting to change the clock on the MDS and OSS to be wildly out of sync with the client and each other, and then run sanity and sanityN (it is still running on a single node, with two mountpoints) to see what fails. That would make it much more obvious where the bad timestamps are coming from.</p> <p>I have 4 VMs each set to a different day of the week.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>t0: -0day = Tue, mdt1 t1: -1day = Mon, mdt2,3,4 t2: -2day = Sun, ost* t3: -3day = Sat, client </pre> </div></div> <p>Here is what I ran on t3 (the client).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">( pdsh -Rssh -w <span class="code-quote">"root@t[0-3]"</span> umount -a -t ldiskfs (export CONFIG=/root/local.sh; source $CONFIG; sh $LUSTRE/tests/llmount.sh) &amp;&gt; /dev/<span class="code-keyword">null</span> echo BEFORE pdsh -Rssh -w <span class="code-quote">"root@t[0-3]"</span> date | sort lfs mkdir -c4 /mnt/lustre/d0 echo AFTER stat /mnt/lustre/d0 pdsh -Rssh -w <span class="code-quote">"root@t[0-3]"</span> date | sort (export CONFIG=/root/local.sh; source $CONFIG; sh $LUSTRE/tests/llmountcleanup.sh) &amp;&gt; /dev/<span class="code-keyword">null</span> ssh t0 insmod /root/lustre-release/ldiskfs/ldiskfs.ko ssh t1 insmod /root/lustre-release/ldiskfs/ldiskfs.ko ssh t0 mkdir -p /tmp/mdt1 ssh t1 mkdir -p /tmp/mdt{2,3,4} ssh t0 mount /tmp/lustre-mdt1 /tmp/mdt1 -o loop,ro -t ldiskfs ssh t1 mount /tmp/lustre-mdt2 /tmp/mdt2 -o loop,ro -t ldiskfs ssh t1 mount /tmp/lustre-mdt3 /tmp/mdt3 -o loop,ro -t ldiskfs ssh t1 mount /tmp/lustre-mdt4 /tmp/mdt4 -o loop,ro -t ldiskfs ssh t0 stat /tmp/mdt1/ROOT/d0 /tmp/mdt1/ROOT/d0/* ssh t1 stat /tmp/mdt{2,3,4}/REMOTE_PARENT_DIR/* ) 2&gt;&amp;1 | tee mkdir.out </pre> </div></div> <p>Here is the edited output. As you would expect after a create, for each inode stated the a,c,m times were equal. So I only show one timestamp per inode.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Host times before the create: t0: Tue Jan 20 13:29:00 CST 2015 t1: Mon Jan 19 13:28:58 CST 2015 t2: Sun Jan 18 13:28:59 CST 2015 t3: Sat Jan 17 13:28:59 CST 2015 /mnt/lustre/d0 2015-01-20 13:29:01.000000000 -0600 # Time seen by client, wrong, comes from t0. /tmp/mdt1/ROOT/d0 2015-01-17 13:28:59.000000000 -0600 # Correct. /tmp/mdt1/ROOT/d0/[0x400000401:0x1:0x0]:0 2015-01-20 13:29:01.139000330 -0600 # Wrong, from t0. /tmp/mdt1/ROOT/d0/[0x440000400:0x1:0x0]:1 2015-01-20 13:29:01.139000330 -0600 # Wrong but does not matter, from t0. /tmp/mdt1/ROOT/d0/[0x480000400:0x1:0x0]:2 2015-01-20 13:29:01.139000330 -0600 # ... /tmp/mdt1/ROOT/d0/[0x4c0000400:0x1:0x0]:3 2015-01-20 13:29:01.139000330 -0600 # ... /tmp/mdt2/REMOTE_PARENT_DIR/0x440000400:0x1:0x0 2015-01-17 13:28:59.000000000 -0600 # Correct. /tmp/mdt3/REMOTE_PARENT_DIR/0x480000400:0x1:0x0 2015-01-17 13:28:59.000000000 -0600 # Correct. /tmp/mdt4/REMOTE_PARENT_DIR/0x4c0000400:0x1:0x0 2015-01-17 13:28:59.000000000 -0600 # Correct. </pre> </div></div> <p>Here is the full output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BEFORE t0: Tue Jan 20 13:29:00 CST 2015 t1: Mon Jan 19 13:28:58 CST 2015 t2: Sun Jan 18 13:28:59 CST 2015 t3: Sat Jan 17 13:28:59 CST 2015 AFTER File: `/mnt/lustre/d0' Size: 16384 Blocks: 8 IO Block: 4096 directory Device: 2c54f966h/743766374d Inode: 288230393331580929 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-20 13:29:01.000000000 -0600 Modify: 2015-01-20 13:29:01.000000000 -0600 Change: 2015-01-20 13:29:01.000000000 -0600 t0: Tue Jan 20 13:29:01 CST 2015 t1: Mon Jan 19 13:28:59 CST 2015 t2: Sun Jan 18 13:29:00 CST 2015 t3: Sat Jan 17 13:29:00 CST 2015 File: `/tmp/mdt1/ROOT/d0' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 62558 Links: 6 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-17 13:28:59.000000000 -0600 Modify: 2015-01-17 13:28:59.000000000 -0600 Change: 2015-01-17 13:28:59.000000000 -0600 File: `/tmp/mdt1/ROOT/d0/[0x400000401:0x1:0x0]:0' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 31302 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-20 13:29:01.139000330 -0600 Modify: 2015-01-20 13:29:01.139000330 -0600 Change: 2015-01-20 13:29:01.139000330 -0600 File: `/tmp/mdt1/ROOT/d0/[0x440000400:0x1:0x0]:1' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 62559 Links: 2 Access: (0000/d---------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-20 13:29:01.139000330 -0600 Modify: 2015-01-20 13:29:01.139000330 -0600 Change: 2015-01-20 13:29:01.139000330 -0600 File: `/tmp/mdt1/ROOT/d0/[0x480000400:0x1:0x0]:2' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 62560 Links: 2 Access: (0000/d---------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-20 13:29:01.139000330 -0600 Modify: 2015-01-20 13:29:01.139000330 -0600 Change: 2015-01-20 13:29:01.139000330 -0600 File: `/tmp/mdt1/ROOT/d0/[0x4c0000400:0x1:0x0]:3' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 62561 Links: 2 Access: (0000/d---------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-20 13:29:01.139000330 -0600 Modify: 2015-01-20 13:29:01.139000330 -0600 Change: 2015-01-20 13:29:01.139000330 -0600 File: `/tmp/mdt2/REMOTE_PARENT_DIR/0x440000400:0x1:0x0' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 700h/1792d Inode: 31302 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-17 13:28:59.000000000 -0600 Modify: 2015-01-17 13:28:59.000000000 -0600 Change: 2015-01-17 13:28:59.000000000 -0600 File: `/tmp/mdt3/REMOTE_PARENT_DIR/0x480000400:0x1:0x0' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 701h/1793d Inode: 31302 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-17 13:28:59.000000000 -0600 Modify: 2015-01-17 13:28:59.000000000 -0600 Change: 2015-01-17 13:28:59.000000000 -0600 File: `/tmp/mdt4/REMOTE_PARENT_DIR/0x4c0000400:0x1:0x0' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 702h/1794d Inode: 62552 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-01-17 13:28:59.000000000 -0600 Modify: 2015-01-17 13:28:59.000000000 -0600 Change: 2015-01-17 13:28:59.000000000 -0600 </pre> </div></div> <p>So we are wrongly using a server timestamp for the stripe 0 agent inode which is affecting the client visible timestamps.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13473" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13473</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6087" title="sanity test_300b: mtime not change after create" class="issue-link" data-issue-key="LU-6087"><del>LU-6087</del></a> lod: use correct attrs in striped directory create<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 518b4efade578cceabcd37f7730c1229889631bc</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13473/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13473/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6087" title="sanity test_300b: mtime not change after create" class="issue-link" data-issue-key="LU-6087"><del>LU-6087</del></a> lod: use correct attrs in striped directory create<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4109062f3d6999353672e1d685b09bf38f9d6c37</p> <p>Patch landed to Master.</p> Duplicate LU-6366 Related LU-6137 LU-6097 Development Rank 1|hzx3c7: Rank (Obsolete) 16943 Severity