Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-5951] sanity test_39k: mtime is lost on close https://jira.whamcloud.com/browse/LU-5951 Lustre <p>This issue was created by maloo for John Hammond <john.hammond@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/01943f56-7150-11e4-b80a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/01943f56-7150-11e4-b80a-5254006e85c2</a>.</p> <p>The sub-test test_39k failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mtime is lost on close: 1416505386, should be 1384969360 </pre> </div></div> <p>I ran 39k in a loop locally and saw the same failure in 2 out of 256 runs.</p> <p>Here are all the instances from maloo:</p> <p><a href="https://testing.hpdd.intel.com/sub_tests/7e8dab70-069d-11e2-9e80-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/7e8dab70-069d-11e2-9e80-52540035b04c</a> ~2012-09-24<br/> <a href="https://testing.hpdd.intel.com/sub_tests/9f92a846-0c4e-11e2-8132-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/9f92a846-0c4e-11e2-8132-52540035b04c</a> ~2012-10-01</p> <p><a href="https://testing.hpdd.intel.com/sub_tests/bed3a506-06c6-11e4-9c81-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/bed3a506-06c6-11e4-9c81-5254006e85c2</a> 2014-07-08 01:42:48 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/127d427e-0c86-11e4-8fe6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/127d427e-0c86-11e4-8fe6-5254006e85c2</a> 2014-07-15 21:43:59 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/ab1c3752-2a76-11e4-8657-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/ab1c3752-2a76-11e4-8657-5254006e85c2</a> 2014-08-23 00:04:17 UTCs</p> <p><a href="https://testing.hpdd.intel.com/sub_tests/1791dfdc-6d01-11e4-8bd3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/1791dfdc-6d01-11e4-8bd3-5254006e85c2</a> 2014-11-14 08:35:54 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/545725c0-6db6-11e4-a728-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/545725c0-6db6-11e4-a728-5254006e85c2</a> 2014-11-15 20:02:55 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/9c2721d8-7078-11e4-a6ba-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/9c2721d8-7078-11e4-a6ba-5254006e85c2</a> 2014-11-19 16:37:26 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/bb99d730-712d-11e4-9495-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/bb99d730-712d-11e4-9495-5254006e85c2</a> 2014-11-20 17:12:53 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/2a99b35e-7150-11e4-b80a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/2a99b35e-7150-11e4-b80a-5254006e85c2</a> 2014-11-20 17:12:53 UTCs<br/> <a href="https://testing.hpdd.intel.com/sub_tests/1db886d4-7177-11e4-89a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/1db886d4-7177-11e4-89a9-5254006e85c2</a> 2014-11-21 00:07:35 UTCs</p> <p>Info required for matching: sanity 39k</p> LU-5951

sanity test_39k: mtime is lost on close

Bug Critical Resolved Fixed Niu Yawei Maloo 22pl Mon, 24 Nov 2014 16:19:06 +0000 Tue, 13 Dec 2016 17:17:23 +0000 Wed, 9 Dec 2015 04:53:57 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.5.4 Lustre 2.8.0 0 16 <p>Just looking back at patches that landed before Nov 14. I found <a href="http://review.whamcloud.com/12243" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12243</a> from <a href="https://jira.whamcloud.com/browse/LU-5006" title="chown/chgrp doesn't work for files created by lfs setstripe" class="issue-link" data-issue-key="LU-5006"><del>LU-5006</del></a> that landed on Nov 11, and it is the only patch that appears related to this part of the code in that time period.</p> <p>Another possibility is <a href="http://review.whamcloud.com/10858" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10858</a> "<a href="https://jira.whamcloud.com/browse/LU-3259" title="cl_lock refactoring" class="issue-link" data-issue-key="LU-3259"><del>LU-3259</del></a> clio: cl_lock simplification" which was also present in the earlier failures on 2014-07-08 and 2014-07-15 on their development branches, and it was landed to master on 2014-11-04.</p> <p>Looks this was introduced when integrating OFD stack:<br/> a67ea1c5</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Author: Mikhail Pershin <tappro@whamcloud.com> Date: Wed May 23 23:00:33 2012 +0400 LU-1406 ofd: IO operations add IO functions to OFD </pre> </div></div> <p>see the ofd_commitrw():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ <span class="code-keyword">if</span> (cmd == OBD_BRW_WRITE) { + /* Don't update timestamps <span class="code-keyword">if</span> <span class="code-keyword">this</span> write is older than a + * setattr which modifies the timestamps. b=10150 */ + + /* XXX when we start having persistent reservations <span class="code-keyword">this</span> needs + * to be changed to ofd_fmd_get() to create the fmd <span class="code-keyword">if</span> it + * doesn't already exist so we can store the reservation handle + * there. */ + valid = OBD_MD_FLUID | OBD_MD_FLGID; + fmd = ofd_fmd_find(exp, &info->fti_fid); + <span class="code-keyword">if</span> (!fmd || fmd->fmd_mactime_xid < info->fti_xid) + valid |= OBD_MD_FLATIME | OBD_MD_FLMTIME | + OBD_MD_FLCTIME; </pre> </div></div> <p>This actually should be:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (fmd && fmd->fmd_mactime_xid > info->fti_xid) valid &=~ time_flags; </pre> </div></div> <p>I'm going to cook a patch soon.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12865" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12865</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ofd: typo in ofd_commitrw()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0c0b41d98d4c9d76082652d637c0e58726a5674b</p> <p>patch for master: <a href="http://review.whamcloud.com/12865" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12865</a></p> <p>While verifying patch <a href="http://review.whamcloud.com/12804" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12804</a> on Lustre b2_5 branch, the same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fdc66684-7176-11e4-89a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fdc66684-7176-11e4-89a9-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12865/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12865/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> clio: update timestamps after buiding rpc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 665ad328b368f1cbd8646690999b609d7b0feaf9</p> <p>More instance on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/65e1bd20-7d90-11e4-8c81-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/65e1bd20-7d90-11e4-8c81-5254006e85c2</a></p> <p>patch landed on master.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13261" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13261</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> clio: update timestamps after buiding rpc<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 7844e1db6d17ae1721c7b1955404ea12bb08b8ad</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13261/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13261/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> clio: update timestamps after buiding rpc<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 7b70c1598a9b484cfe7f50c584caaca5ab64f0ba</p> <p>It seems a regression, I saw it twice recently</p> <p><a href="https://testing.hpdd.intel.com/sub_tests/3032df32-1fbc-11e5-bc94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/3032df32-1fbc-11e5-bc94-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/7e4a1dbe-1fe1-11e5-9b0e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/7e4a1dbe-1fe1-11e5-9b0e-5254006e85c2</a></p> <p>I think the regression should be introduced by: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">commit bf3e7f67cb33f3b4e0590ef8af3843ac53d0a4e8 Author: Gregoire Pichon <gregoire.pichon@bull.net> Date: Wed May 13 16:42:44 2015 +0200 LU-5319 ptlrpc: embed highest XID in each request Atomically assign XIDs and put request and sending list so we can learn the lowest unreplied XID at any point. This allows to embed in every resquests the highest XID <span class="code-keyword">for</span> which a reply has been received and does not have an unreplied lower-numbered XID. This will be used by the MDT target to release in-memory reply data corresponding to XIDs of reply received by the client. Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com> Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net> Change-Id: Ic88fb6db704d8e9a78a34fe16f64abb2cdffc4c4 Reviewed-on: http:<span class="code-comment">//review.whamcloud.com/14793 </span> Tested-by: Jenkins Tested-by: Maloo <hpdd-maloo@intel.com> Reviewed-by: Andreas Dilger <andreas.dilger@intel.com> Reviewed-by: Oleg Drokin <oleg.drokin@intel.com> </pre> </div></div> <p>Where we deferred the xid assignment from request packing to request sending, that breaks the fix of bug 10150, see osc_build_rpc():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* Need to update the timestamps after the request is built in <span class="code-keyword">case</span> * we race with setattr (locally or in queue at OST). If OST gets * later setattr before earlier BRW (as determined by the request xid), * the OST will not use BRW timestamps. Sadly, there is no obvious * way to <span class="code-keyword">do</span> <span class="code-keyword">this</span> in a single call. bug 10150 */ </pre> </div></div> <p>Looks we have to fix the race of setattr vs. brw in another method or just fix the multi-slot patch, any suggestions?</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15473" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15473</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> osc: set ioepoch to ost setattr/punch/write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fafc374824db7d69bed1c527989ea60d825200dd</p> <p>It isn't totally clear that we need the change from <a href="http://review.whamcloud.com/14793" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14793</a> in order for the multi-slot code to work. While it would make the tracking of unreplied RPCs a bit more complex, having an atomic XID assignment set at "send" time is not quite the same as "unreplied" so there still needs to be a mechanism used to track which RPCs have replies.</p> <p>The one major difference would be that there needs to be some mechanism to track RPC XIDs which are never sent, so that they don't permanently get stuck as the lowest unreplied XID. It would seem possible to do this in __ptlrpc_req_free() I think?</p> <p>well, if we don't track that, then it's very easy to "lose" some slots: at moment X we used 8 slots, then later we were using 2 slots at most. using tags we can reuse only those 2 slots, but we can't report the others slots can be reused. there is no strong need to maintain that absolutely up to date,<br/> technically it should be possible (and not very complex) to introduce another list, like.. ptlrpc_next_xid() (or it's callers) atomically puts RPC on the list, after_reply() and __ptlrpc_req_free() delete the RPC from the list.</p> <p>Ok, I'll update the patch to maintain an unreplied xid list for each import.</p> <p>I've seen this issue again:<br/> 2015-07-02 18:54:42 - <a href="https://testing.hpdd.intel.com/test_sets/4ee3283c-2102-11e5-8eb6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4ee3283c-2102-11e5-8eb6-5254006e85c2</a><br/> 2015-07-03 18:45:11 - <a href="https://testing.hpdd.intel.com/test_sets/f4ce726e-21e9-11e5-a388-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f4ce726e-21e9-11e5-a388-5254006e85c2</a><br/> 2015-07-10 05:07:18 - <a href="https://testing.hpdd.intel.com/test_sets/0d7b77d2-26fc-11e5-925d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0d7b77d2-26fc-11e5-925d-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15473/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15473/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ptlrpc: track unreplied requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c77e504fdac12d3be7d19a652d6c7da497018c76</p> <p>Landed for 2.8.0</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16734" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16734</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ptlrpc: track unreplied requests"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e12c29a48b33a0ae7bd4147dab57dae5597954aa</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16734/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16734/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ptlrpc: track unreplied requests"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c250f40ef3222dbeb92d7914a0d9f38a3525d2fb</p> <p>Reopening as the recent landing had caused <a href="https://jira.whamcloud.com/browse/LU-7252" title="replay-single test_116b:process_req_last_xid()) LBUG" class="issue-link" data-issue-key="LU-7252"><del>LU-7252</del></a>. The recent landing has been reverted.</p> <p>The fixVersion has been updated to 2.9.0 to properly address the issue that was being addressed by <a href="http://review.whamcloud.com/15473/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15473/</a><br/> This issue is not something we are currently seeing as a failure on master.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16759" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16759</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ptlrpc: track unreplied requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c4af36bfdb61fdbdf7d43778859d3e8b441531b5</p> <p>The patch (<a href="http://review.whamcloud.com/#/c/16759/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16759/</a>) here is necessary for GSS Shared Key (and I assume Kerberos) to function without generating an LBUG.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16759/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16759/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> ptlrpc: track unreplied requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8c69ef1e5caf3b6800e83bb73696b4bd1ae6e613</p> <p>Landed for 2.8</p> Blocker LU-3289 Duplicate Related LU-5006 LU-5319 LU-6831 LU-7252 LU-7182 Development Rank 1|hzx1fb: Rank (Obsolete) 16613 Severity