https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14765 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/8d692dc1-910c-4ee7-baf0-21734795ed81" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8d692dc1-910c-4ee7-baf0-21734795ed81</a></p> <p>test_44c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mirror split does not reduce block# 10485760 != 4194304 </pre> </div></div> <p>sanity-flr test_44c seems buggy. It starts by writing 10MB, but it reports a file size of 4MB:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>** before mirror ops, file blocks=4096 KiB </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-flr test_44c - mirror split does not reduce block# 10485760 != 4194304</p> LU-14765

sanity-flr test_44c: mirror split does not reduce block#

Bug Major Open Unresolved Zhenyu Xu Maloo always_except Wed, 16 Jun 2021 06:07:21 +0000 Fri, 14 Apr 2023 23:57:44 +0000 Lustre 2.15.0 0 10 <p>Test was added by patch <a href="https://review.whamcloud.com/43168" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43168</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14526" title="SoM xattr not updated after mirror split" class="issue-link" data-issue-key="LU-14526"><del>LU-14526</del></a> flr: mirror split downgrade SOM</tt>".  </p> <p>maybe a short sleep is needed before fetching the first attributes?</p> <p>Bobijam, can you please take a look. </p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/44074" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44074</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14765" title="sanity-flr test_44c: mirror split does not reduce block#" class="issue-link" data-issue-key="LU-14765">LU-14765</a> test: sanity-flr/44c sleep before collect block size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 93808f4e4c682db5727f392ff477aff82ca6d7e3</p> <p>+1 on master<br/> <a href="https://testing.whamcloud.com/test_sets/94e73271-fa7a-495a-9352-5fab56b49dd0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/94e73271-fa7a-495a-9352-5fab56b49dd0</a></p> <p>+2 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/3f1b28ed-a2fe-4493-a883-b3c6130b1031" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3f1b28ed-a2fe-4493-a883-b3c6130b1031</a><br/> <a href="https://testing.whamcloud.com/test_sets/8be2ef78-5e26-49d7-8a4a-22a5c4b1fa02" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8be2ef78-5e26-49d7-8a4a-22a5c4b1fa02</a></p> <p>+1<br/> <a href="https://testing.whamcloud.com/test_sessions/7669955f-7036-4df9-8551-5fd23a70d5c6" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/7669955f-7036-4df9-8551-5fd23a70d5c6</a></p> <p>The block number of a mirrored file cannot be accurate IMO, since getattr of the file only retrieve info from the OST objects of one mirror, not including those of another mirrors. </p> <p>And mirror extend can only add block numbers of one mirror upon the already inaccurate block numbers of the file; and if multiple mirror extension is executed by "-N &lt;# of mirrors&gt;" option, the situation is exacerbated.</p> <p>So I'm wondering whether we need to keep the semantic meaning of block # in a mirrored file.</p> <p>Shouldn't "stat" be using the blocks in the SOM xattr, and that should be kept correct? I would rather ensure this kept updated for the few times that the SOM is put in "strict" mode, so that incremental mirror addition is also correct.</p> <p>For this test, it might make sense to have it <b>only</b> use "<tt>lfs getsom -b</tt>" to verify the SOM data directly. In order for that to work, I also think that the <tt>trusted.som</tt> xattr never be cached on the client.</p> <p>I don't see that the inode's block number (the number reported by "stat") corresponds to the SOM block number after several mirror extend calls, even SOM is in "strict" mode. Updated SOM is not always passed back and translated to inode's block number. </p> <p>i would think that when SOM sets the STRICT flag that all client writes are complete and committed, so the blocks could <em>should</em> be correct at this time? Alternately, should we allow the SOM blocks count to be increased from later client RPCs, like LSOM does, so that the <em>size</em> is correct/strict and the blocks at least get "more correct" over time?</p> <p>yes, write do account correct block number changes, but a file with multiple mirrors will mess the block number base (so I'm thinking even mirror extend need to invalidate the block number in SOM). </p> <p>And its block number cannot be correct until it is split to remain only one mirror.</p> <p>Bobijam, is there something unusual with how the SOM blocks count is updated? I'd think that it is just using the sum of the blocks counts for all mirror objects at the time the file is closed. If some of the mirrors are stale, then blocks count may not be totally accurate, but after "<tt>lfs mirror resync</tt>" the client doing the resync should be totally uptodate and can send this to the MDS.</p> <p>+1 on master<br/> <a href="https://testing.whamcloud.com/test_sets/950a96fc-10c7-41a8-b7d2-a8b14834b02b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/950a96fc-10c7-41a8-b7d2-a8b14834b02b</a></p> <blockquote><p>Bobijam, is there something unusual with how the SOM blocks count is updated? I'd think that it is just using the sum of the blocks counts for all mirror objects at the time the file is closed. If some of the mirrors are stale, then blocks count may not be totally accurate, but after "lfs mirror resync" the client doing the resync should be totally uptodate and can send this to the MDS.</p></blockquote> <p>The initial issues raised is that 'stat' does not show file blocks number increase after it has extra mirror merged.</p> <p>If one mirror is added, supposed that SOM is strict and it reflects all blocks of the file, then we can account the blocks number accurately after the merge; if the SOM is not strict, then after the merge the SOM is still not accurate, then 'stat' need to rely on glimpse size to collect the file's blocks number, but glimpse would only choose one mirror to calculate blocks number for one mirror, and the result blocks number would not be bigger than that of the pre-mirror-extended file.</p> <p>The similar scenarios also happens on mirror split, if SOM is not accurate on the file's blocks number before the mirror operation, the blocks number would not be showed to be decreased after the mirror split as well.</p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/3d97f947-b99d-4316-9b42-6f2502a96613" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3d97f947-b99d-4316-9b42-6f2502a96613</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/d5df04c7-cfe9-4a8e-99a9-e7b2beb3f55b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d5df04c7-cfe9-4a8e-99a9-e7b2beb3f55b</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44074/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44074/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14765" title="sanity-flr test_44c: mirror split does not reduce block#" class="issue-link" data-issue-key="LU-14765">LU-14765</a> test: disable sanity-flr/44c<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a45fe93cd8d1e941d58e0f11e21649e1956ba2c7</p> <p>This is causing user-visible issues in the field, since "du" with STRICT SOM reports block numbers that are too large after the mirror has been split from the file. There was some work in <a href="https://jira.whamcloud.com/browse/LU-14526" title="SoM xattr not updated after mirror split" class="issue-link" data-issue-key="LU-14526"><del>LU-14526</del></a> to reset the blocks count for SOM files on split, but for STRICT SOM the values should actually be correct, since they are returned to userspace as authoritative values. </p> <p>Rather than have the client do an extra "stat" of all mirrors on each close (which is expensive and mostly useless), I think the right answer is that this extra stat is only needed when doing the mirror split operation. That will ensure the right values are stored in the STRICT SOM, without adding overhead to the very common close operation. </p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50310" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50310</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14765" title="sanity-flr test_44c: mirror split does not reduce block#" class="issue-link" data-issue-key="LU-14765">LU-14765</a> test: enable sanity-flr/44c<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c945ba741e73eb42c0d013d2a1ab44bda416986b</p> Related LU-14526 LU-15876 LU-16465 Development Rank 1|i01x13: Rank (Obsolete) 9223372036854775807 Severity