https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7310 Lustre <p>sanityn test 39a fails only on review-zfs-part-1 with the error message </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>'mtime is not updated on write: 1444890978 &lt;= 1413354977 &lt;= 1444890979' </pre> </div></div> <p>Logs for the most recent failure is at <a href="https://testing.hpdd.intel.com/test_sets/560c385e-7325-11e5-ae2e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/560c385e-7325-11e5-ae2e-5254006e85c2</a>. </p> <p>sanityn test_39a has failed with this error about 18 times in the past two months only on review-zfs-part-1. All of these failures took place when testing patches for <a href="https://jira.whamcloud.com/browse/LU-7108" title="Remove sanityn tests 14b, 19, 29, and 35 from the ALWAYS_EXCEPT list" class="issue-link" data-issue-key="LU-7108"><del>LU-7108</del></a> or as a results of <a href="https://jira.whamcloud.com/browse/LU-7072" title="sanityn test_78: Expected set_param to return 0 or EAGAIN" class="issue-link" data-issue-key="LU-7072"><del>LU-7072</del></a> with patches <a href="http://review.whamcloud.com/#/c/16383" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16383</a> and <a href="http://review.whamcloud.com/#/c/16164" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16164</a>, respectively. Both of these patches remove tests from the sanityn ALWAYS_EXCEPT list; sanityn tests 14b, 19, 29, and 35. The patch for <a href="https://jira.whamcloud.com/browse/LU-7072" title="sanityn test_78: Expected set_param to return 0 or EAGAIN" class="issue-link" data-issue-key="LU-7072"><del>LU-7072</del></a> added sanityn test_78 to the ALWAYS_EXCEPT list and, by accident, removed all other tests from the except list allowing these test to run. More recently, a patch for <a href="https://jira.whamcloud.com/browse/LU-7108" title="Remove sanityn tests 14b, 19, 29, and 35 from the ALWAYS_EXCEPT list" class="issue-link" data-issue-key="LU-7108"><del>LU-7108</del></a>, removing sanityn tests 14b, 19, 29, and 35 from the ALWAYS_EXCEPT list is working its way through autotest. What both of these tickets/patch have in common is that their autotest results show that by reintroducing these sanityn tests back into the mix, they are probably causing test 39a to occasionally fail.</p> <p>Recent logs with sanityn_test39a failures are at<br/> 2015-10-10 10:13:27 - <a href="https://testing.hpdd.intel.com/test_sets/1ef9bb16-6f56-11e5-b7e0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1ef9bb16-6f56-11e5-b7e0-5254006e85c2</a><br/> 2015-10-06 04:23:35 - <a href="https://testing.hpdd.intel.com/test_sets/85fef54c-6c09-11e5-9ae6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/85fef54c-6c09-11e5-9ae6-5254006e85c2</a><br/> 2015-09-22 10:55:32 - <a href="https://testing.hpdd.intel.com/test_sets/5ec3968e-6139-11e5-b495-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ec3968e-6139-11e5-b495-5254006e85c2</a><br/> 2015-09-17 23:36:18 - <a href="https://testing.hpdd.intel.com/test_sets/989a1506-5dd1-11e5-a637-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/989a1506-5dd1-11e5-a637-5254006e85c2</a><br/> 2015-09-12 10:40:34 - <a href="https://testing.hpdd.intel.com/test_sets/170cc7ba-596a-11e5-88c5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/170cc7ba-596a-11e5-88c5-5254006e85c2</a><br/> 2015-09-05 04:32:19 - <a href="https://testing.hpdd.intel.com/test_sets/89465b6c-53ba-11e5-83ee-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/89465b6c-53ba-11e5-83ee-5254006e85c2</a></p> autotest review-zfs-part-1 LU-7310

sanityn test_39a fails with 'mtime is not updated on write: 1444890978 <= 1413354977 <= 1444890979'

Bug Minor Resolved Fixed WC Triage James Nunez Fri, 16 Oct 2015 18:27:35 +0000 Tue, 13 Sep 2016 20:17:37 +0000 Tue, 13 Sep 2016 20:17:37 +0000 Lustre 2.8.0 Lustre 2.9.0 0 6 <p>I suspect that the enabling of these tests may cause the server VM to be slowed down enough cause the clock to skew, and there is a bug in the server code updating some timestamp that uses the server clock instead of the timestamp passed from the client.</p> <p>This could be exposed by running this test with a wildly different time set on the MDS and OSS (e.g. ahead or behind by 1 day) so that it is clear where the timestamp is coming from. This was partially done in <a href="http://review.whamcloud.com/14122" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14122</a> but needs to be done in a better manner to be able to include into our regular testing.</p> <p>I ran some manual test on single node with various combinations of 14b, 19, 29, 35 in ALWAYS_EXCEPT list and found out whenever test_35 was excluded from ALWAYS_EXCEPT list it produced above mentioned error for test_39a. So there is a high possibility of this issue being related to test_35 in someway.</p> <p>Another failure:<br/> 2016-01-25 22:00:44 - <a href="https://testing.hpdd.intel.com/test_sets/69450134-c3cb-11e5-b763-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/69450134-c3cb-11e5-b763-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/9f0e675e-3920-11e6-bbf5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f0e675e-3920-11e6-bbf5-5254006e85c2</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00010000:00010000:0.0:1444890979.493775:0:30329:0:(ldlm_lockd.c:1923:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff880037e41c00 lock: ffff88007a6fda80/0x4e64250580a5fff6 lrc: 2/0,0 mode: PW/PW res: [0x7f8f:0x0:0x0].0x0 rrc: 1 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;18446744073709551615) flags: 0x20000020000 nid: local remote: 0x57fb8313db314742 expref: -99 pid: 1011 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1444890979.493817:0:30329:0:(cl_object.c:290:cl_object_glimpse()) header@ffff88005e632978[0x0, 2, [0x200002b10:0x5e:0x0] hash] 00000020:00010000:0.0:1444890979.493821:0:30329:0:(cl_object.c:290:cl_object_glimpse()) size: 6 mtime: 1413354977 atime: 1444890978 ctime: 1444890978 blocks: 1 00000020:00010000:0.0:1444890979.493828:0:30329:0:(cl_object.c:290:cl_object_glimpse()) header@ffff88007a767090[0x0, 2, [0x100000000:0x7f8f:0x0] hash] 00000020:00010000:0.0:1444890979.493830:0:30329:0:(cl_object.c:290:cl_object_glimpse()) size: 6 mtime: 1413354977 atime: 1444890978 ctime: 1444890978 blocks: 1 00000100:00100000:0.0:1444890979.493881:0:30329:0:(service.c:2121:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ldlm_cb00_000:LOV_OSC_UUID+4:24810:x1515071670949744:12345-10.1.5.15@tcp:106 Request procesed in 125us (196us total) trans 0 rc 0/0 </pre> </div></div> <p>The client appending file returns wrong mtime on glimpse, looks some defect in clio that can only be triggered in zfs testing?</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21063" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21063</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7310" title="sanityn test_39a fails with &#39;mtime is not updated on write: 1444890978 &lt;= 1413354977 &lt;= 1444890979&#39;" class="issue-link" data-issue-key="LU-7310"><del>LU-7310</del></a> clio: sync write should update mtime<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d9ebc5916afad38629def87e17e0e363a9d5c027</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21063/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21063/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7310" title="sanityn test_39a fails with &#39;mtime is not updated on write: 1444890978 &lt;= 1413354977 &lt;= 1444890979&#39;" class="issue-link" data-issue-key="LU-7310"><del>LU-7310</del></a> clio: sync write should update mtime<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e130c75827b2f40d89cb69ac5c64bb94d67c520c</p> <p>Landed for 2.9</p> Related LU-6137 LU-7072 LU-7108 Development Rank 1|hzxqp3: Rank (Obsolete) 9223372036854775807 Severity