https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13602 Lustre <p>Currently, in <tt>lov_init_composite()</tt> will quit with error when reading an unknown LOV pattern:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> CERROR("%s: unknown composite layout entry type %i\n", lov2obd(dev-&gt;ld_lov)-&gt;obd_name, lsm-&gt;lsm_entries[i]-&gt;lsme_pattern); </pre> </div></div> <p>Since FLR will be used for upcoming new features, like PCC-RO, an old client should still be able to read the old format mirrors and ignore the new types of mirrors.</p> <p>On the MDS, it should skip mirrors with unknown/foreign component types and mark them stale when selecting which component to use for writes.</p> LU-13602

Skip unknown FLR component types

Bug Critical Resolved Fixed Qian Yingjin Li Xi Wed, 27 May 2020 01:40:06 +0000 Mon, 2 Aug 2021 17:55:27 +0000 Thu, 29 Jul 2021 11:42:41 +0000 Lustre 2.15.0 0 7 <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39513" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39513</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13602" title="Skip unknown FLR component types" class="issue-link" data-issue-key="LU-13602"><del>LU-13602</del></a> flr: skip unknown FLR component types<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3bb436e0172f1a2e3a0fa4806cf15f3c1d9682e3</p> <p>Yingjin, could you please add a short description of what you think is needed to implement this.</p> <p>To my thinking, the main places that need to be changed are mirror selection for writes (on the MDS) and reads (on the client) to skip unknown component types (e.g. unknown layout types, foreign components, PCC/HSM components on remote nodes, etc) in addition to the normal skip of STALE components. For PCC-RO the client can select the local cache copy if it is not STALE.</p> <p>Is there some other change that is needed here which I'm missing?</p> <p>Hi Andreas,</p> <p>The problem is not PCC-RW as it is actual a HSM solution and the file data can be restored when a conflict access happens.</p> <p>The main problem is about PCC-RO. When a file is PCC-RO cached on a client, the components on Lustre OSTs and on the PCC copies are both in update-to-date state (not STALE state). All client can do read operations on the components with data on Lustre OSTs.<br/> We use layout lock to protect the validity for PCC read-only caching services.<br/> When a file is PCC-RO cached on a client, the layout on the MDT is flagged with <em>L.rdonly</em> state. When a client modifies this file <em>L.rdonly</em> state, it must clear the read-only state on MDT with an intent lock request with a write layout intent. When MDT received this write layout intent request, it will acquire EX layout lock on the file which will revoke all layout locks on clients and invalidate the cached layouts.<br/> However, for a old client without PCC-RO support, it can not recognize that the file is in PCC-RO state after the server grants the layout lock to this client, thus it can directly write to the component with data on Lustre OSTs without clearing the <em>L.rdonly</em> state on the MDT. In this case, some client will continue to read data from the local PCC-RO copy while the old client is writing the file content, this results in inconsistent access.</p> <p>To solve this problem, the solutions may be:</p> <ul class="alternate" type="square"> <li>Add a PCC-RO connection flag;</li> <li>When a client without PCC-RO support requests a layout lock on the file that is in <em>L.rdonly</em> state on MDT, the MDT clears the <em>L.rdonly</em> flag on the layout first (this maybe invalidate all PCC-RO cached copies on clients), and then return the layout to the old client?</li> </ul> <p>Regards,<br/> Qian</p> <p>When you write "for a old client without PCC-RO support", shouldn't the normal FLR handling cover this condition? We already do not allow clients that do not understand FLR from accessing files with mirror components, and PCC-RO should be handled in the same way as any other FLR mirror. The main difference should be that a PCC-RO client has the added option of reading the file from the local cache copy (if the file exists there). Any non-PCC client can only read from the non-PCC mirror(s) of the file, but there should <b>always</b> be a non-PCC mirror for PCC-RO. I can't see any reason why a non-PCC client <b>reading</b> from a file should cause the PCC-RO cache copies to be invalidated?</p> <p>It should only be <b>writing</b> to a FLR file that should cause the layout to be invalidated, and all but one of the mirrors to be marked stale. This should already be handled by normal FLR mechanisms to revoke the layout lock from the clients (including PCC-RO clients), and the PCC client should only be able to access the local cache copy again when it has a layout lock, and the data version of the cache copy is still valid (i.e. the data was not modified). It may be that the PCC code needs to be changed so that it computes and stores the "data version" of <b>only</b> the non-PCC mirror copy instead of the whole layout, so that the PCC copy does not get invalidated just because another mirror was added, or similar.</p> <p>Combining with FLR, we need to extend the data structure `enum lov_comp_md_flags`.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /** * on-disk data <span class="code-keyword">for</span> lcm_flags. Valid <span class="code-keyword">if</span> lcm_magic is LOV_MAGIC_COMP_V1. */ <span class="code-keyword">enum</span> lov_comp_md_flags { <span class="code-comment">/* the least 2 bits are used by FLR to record file state */</span> LCM_FL_NONE = 0x0, LCM_FL_RDONLY = 0x1, LCM_FL_WRITE_PENDING = 0x2, LCM_FL_SYNC_PENDING = 0x4, LCM_FL_PCC_RDONLY = ox8, <span class="code-comment">/* The file is RO-PCC cached */</span> LCM_FL_FLR_MASK = 0xF, }; </pre> </div></div> <p>Please note the flags LCM_FL_PCC_RDONLY is different from LCM_FL_RDONLY here.<br/> A PCC-RO cached file could be in the state:</p> <ul class="alternate" type="square"> <li>LCM_FL_PCC_RDONLY | LCM_FL_RDONLY: it means that all components are synced and in up-to-date state, and then one client attaches the file into the PCC backend FS.</li> <li>LCM_FL_PCC_RDONLY | LCM_FL_WRITE_PENDING: It means the file was once modified by a client, the data content of all components are not synced. The MDT has already picked a primary replication and marked other components as STALE. At this time, a client can still PCC-RO attach the file. For this client, the primary component and the PCC copy are both in up-to-date state.</li> <li>...</li> </ul> <p>From my own opinion, I do not think PCC-RO attaching a file in LCM_FL_WRITE_PENDING state needs to sync all mirrors and transmit the file into LCM_FL_RDONLY state which may be time-consuming. In our implementation, we only need to add a new PCC-RO foreign layout component and add LCM_FL_PCC_RDONLY into the FLR flags.</p> <p>As we add a new LCM_FL_PCC_RDONLY state, thus the old client may not understand this new FLR component flag, and may result in inconsistent access.</p> <p>Regards,<br/> Qian</p> <p>The old on-disk data for lcm_flags are:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /** * on-disk data <span class="code-keyword">for</span> lcm_flags. Valid <span class="code-keyword">if</span> lcm_magic is LOV_MAGIC_COMP_V1. */ <span class="code-keyword">enum</span> lov_comp_md_flags { <span class="code-comment">/* the least 2 bits are used by FLR to record file state */</span> LCM_FL_NONE = 0, LCM_FL_RDONLY = 1, LCM_FL_WRITE_PENDING = 2, LCM_FL_SYNC_PENDING = 3, LCM_FL_FLR_MASK = 0x3, }; </pre> </div></div> <p>We have also changed the value of LCM_FL_SYNC_PENDING...</p> <p>Maybe the new <em>lov_comp_md_flags</em> should be:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /** * on-disk data <span class="code-keyword">for</span> lcm_flags. Valid <span class="code-keyword">if</span> lcm_magic is LOV_MAGIC_COMP_V1. */ <span class="code-keyword">enum</span> lov_comp_md_flags { <span class="code-comment">/* the least 2 bits are used by FLR to record file state */</span> LCM_FL_NONE = 0x00, LCM_FL_RDONLY = 0x01, LCM_FL_WRITE_PENDING = 0x02, LCM_FL_SYNC_PENDING = 0x03, LCM_FL_PCC_RDONLY = 0x10, LCM_FL_FLR_MASK = 0x13, }; </pre> </div></div> <p>we don't change the old FLR component flags, add a new LCM_FL_PCC_RDONLY from 0x10.</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/40791" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40791</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10499" title="Readonly Persistent Client Cache support" class="issue-link" data-issue-key="LU-10499"><del>LU-10499</del></a> pcc: introducing OBD_CONNECT2_PCCRO flag<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d2638ccd5c1a327840c68d875eb30a0a3f4f895d</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/40813" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40813</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13602" title="Skip unknown FLR component types" class="issue-link" data-issue-key="LU-13602"><del>LU-13602</del></a> pcc: add LCM_FL_PCC_RDONLY layout flag<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2dc8af64b58ab5cc6d3ae4964ceda4271d2baef4</p> <p>Yingjin, neither of these patches are addressing the main issue of this ticket, which is to allow the MDS and clients to skip layout types that they do not understand. In particular, in the new PCC code, it is using a foreign layout as part of a file, and if that file is being modified by a client then the PCC component needs to be marked STALE. The same should be done for any other layout type that the MDS does not understand.</p> <p>On the client, it should not try to read from a component type that it doesn't understand. </p> <p>Updated the patch <a href="https://review.whamcloud.com/#/c/39513/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39513/</a>.</p> <p>It will skip the unknown FOREIGN layout component that the client does not support or understand.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39513/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39513/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13602" title="Skip unknown FLR component types" class="issue-link" data-issue-key="LU-13602"><del>LU-13602</del></a> flr: skip unknown FLR component types<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 61a002cd8631ecd35a0326dba68f0eb6e48dc44f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40813/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40813/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13602" title="Skip unknown FLR component types" class="issue-link" data-issue-key="LU-13602"><del>LU-13602</del></a> pcc: add LCM_FL_PCC_RDONLY layout flag<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: adc1bbbf20e0a8a53274aa4590ed0935f954d1bc</p> <p>Landed for 2.15</p> <p>For some reason the change in the patch LCM_FL_PCC_RDONLY exposes a bug that I have seen with the port to the native Linux client. With the patch I get:</p> <p>Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================</p> <p>======= 10:58:33 (1627916313)</p> <p>Lustre: *** cfs_fail_loc=406, val=0***</p> <p>LustreError: 20036:0:(osc_request.c:2586:osc_build_rpc()) prep_req failed: -12</p> <p>LustreError: 20036:0:(osc_cache.c:2259:osc_check_rpcs()) Write request failed with -12</p> <p> LustreError: 20042:0:(lu_object.c:215:lu_object_put()) ASSERTION( list_empty(&amp;top-&gt;loh_lru) ) failed:</p> <p>LustreError: 20042:0:(lu_object.c:215:lu_object_put()) LBUG</p> <p>kernel: Pid: 20042, comm: lctl 5.7.0-rc7+ #1 SMP PREEMPT Sat Jul 31 13:04:46 EDT 2021</p> <p>kernel: Call Trace:</p> <p>kernel: libcfs_call_trace+0x62/0x80 <span class="error">&#91;libcfs&#93;</span></p> <p>kernel: lbug_with_loc+0x41/0xa0 <span class="error">&#91;libcfs&#93;</span></p> <p>kernel: lu_object_put+0x31b/0x340 <span class="error">&#91;obdclass&#93;</span></p> <p>kernel: vvp_pgcache_current+0x7e/0x150 <span class="error">&#91;lustre&#93;</span></p> <p>kernel: seq_read+0x200/0x3c0</p> <p>kernel: full_proxy_read+0x4d/0x70</p> <p>kernel: vfs_read+0xb3/0x17</p> <p>kernel: ksys_read+0x5f/0xd0</p> <p>kernel: do_syscall_64+0x6d/0x4f0</p> <p>kernel: entry_SYSCALL_64_after_hwframe+0x49/0xb3</p> <p>kernel: Kernel panic - not syncing: LBUG</p> Related LU-13637 LU-10499 LU-14319 LU-14389 Development Rank 1|i0119r: Rank (Obsolete) 9223372036854775807 Severity