https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12025 Lustre <p>During OST addition, <br/> 1. MDT gets configuration refreshment first.<br/> 2. Then MDT get create request from the client, might allocate the object the new OST, then reply to the client.<br/> 3. If client does not refresh its configuration yet, then do I/O with the EA, it might get EIO because it does not know the OST.</p> <p>So it probably needs add version to the config log to avoid this happen.</p> <p>This happens quite often in cloud environment. though not sure if there is duplicate ticket already.</p> LU-12025

Adding OST may cause EIO - delay activation of new OSTs on existing filesystem

Bug Minor Resolved Fixed Andreas Dilger Di Wang Wed, 27 Feb 2019 03:34:02 +0000 Sun, 19 Nov 2023 00:24:42 +0000 Fri, 8 Nov 2019 07:56:00 +0000 Lustre 2.13.0 Lustre 2.12.4 0 7 <p>I saw another related comment recently about the desire to be able to configure new OSTs on the filesystem, but not have them immediately active on the MDS until the administrator wants them enabled.</p> <p>I suspect there are a number of simple mechanisms that might allow this to happen:</p> <ul class="alternate" type="square"> <li>have a time delay between when the OST is first added to the filesystem before it can be used for object allocation, maybe 30-60s IFF the OSTs are added and there are clients already mounted and the MDS is not <em>just</em> newly formatted. I suspect we already have workarounds for this issue in <tt>test-framework.sh</tt> (e.g. sleep for some time until the OST is visible on the client) because we are constantly formatting and adding new OSTs to the filesystem, but possibly we are mounting the client after the OSTs and don't see it. There are probably a <em>few</em> places where an OST is added to a running filesystem during a test, but maybe we don't try to write to it immediately.</li> <li>pass a flag from the OST at the first connect time to the MDS that indicates the OST should be inactive until the admin enables it manually. This may be desirable for a number of reasons. This could be done by setting <tt>osp.&lt;fsname&gt;-OSTnnnn.max_create_count=0</tt> on the MDS for the new OSTnnnn the first time it is connected.</li> <li>mark the OST unavailable locally (e.g. <tt>OS_STATE_ENOINO</tt> or something set via <tt>mkfs.lustre</tt>) <em>before</em> it mounts and tries to connect to the MDS the first time so there is no window on the MDS when there is a problem, and it keeps the implementation more localized instead of depending on the MDS as well</li> </ul> <p>Allowing both modes (autmatic activation after timeout, wait for manual activation) may be useful under different circumstances.</p> <p>My initial thought is MGS maintains a version number, which can bump up when new target is added (or removed?), then all other nodes will get the version number when it fetches the config log from MGS. when client send request to server, if their version number does not match, then either client or server will needs to refresh their configuration from MGS. </p> <p>Or even further, each server target can also maintain its own version number (when it is being added to MGS), then the target can process the request as long as the req version number is newer than the target version.</p> <p>It isn't clear that having a per-target version number would help. The target being added (OST) can provide a version number, but the client shouldn't have to pass its "connect/config version" for every OST it knows about to the MDS for every file it is creating.</p> <p>Sending a single "config record version" (really just the last MGS config record number that the client processed) from the client to the MDS with each create would be more useful, since this would (indirectly) tell the MDS which OSTs the client is connected to and it could skip ones that were added after that version. Something like storing a "minimum config record number" on each target in LOD which is the config llog record in which the OST was added, and the client request would include their "current config record number" along with each request. A check like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (req-&gt;current_config_rec &lt; lod-&gt;target[ost_idx]-&gt;tgt_min_config_rec) <span class="code-keyword">continue</span>; </pre> </div></div> <p>during create would be enough to skip the OST for that create.</p> <p><b>HOWEVER</b> this has some significant drawbacks:</p> <ul class="alternate" type="square"> <li>it needs a protocol change so that clients will always send this field with each create request and the MDS will check it, to fix a problem that happens very rarely for most users</li> <li>the config llog record numbers <em>may</em> not be easily accessible in the right parts of the code (haven't looked at that yet)</li> <li>it would only fix the problem of <b>one</b> client creating and using a file <b>itself</b>, but would not fix the problem of a <b>different</b> client trying to access that file before it had processed the config llog updates (which may be delayed tens of seconds if there are many clients)</li> </ul> <p>So, my suggestion for a simpler solution is just to use a timeout (maybe variable), based on how quickly config llog records are processed by clients, before an OST (or MDT, for DNE) can be used for new file allocations. There is already a small delay before the OST could be used, because the MDS needs to precreate objects there, but that is only a fraction of a second in most cases. Instead of storing the "config version" in the LOD target, store the "config time" for the target, and skip it for new allocations for e.g. 10s after it connects. This should handle the case where the MDS itself was just mounted and all OSTs are pre-existing (e.g. only delay usage if the <tt>lov_objids</tt> entry was just added).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>it would only fix the problem of one client creating and using a file itself, but would not fix the problem of a different client trying to access that file before it had processed the config llog updates (which may be delayed tens of seconds if there are many clients) </pre> </div></div> <p>Oh, I mean the stale client should not be allowed to access the "newer" server until it refreshes its own config from MGS, but this indeed needs a lot changes, probably not worth for this as you said. Adding timeout is probably good enough here. thanks.</p> <p>In relation to <a href="https://jira.whamcloud.com/browse/LU-11963" title="Add nonrotational flag to obd_statfs" class="issue-link" data-issue-key="LU-11963"><del>LU-11963</del></a>, one easy way to allow a new OST to be added to the filesystem that would still prevent the MDS from using it immediately would be to send the "<tt>OS_STATE_NOPRECREATE</tt>" flag from the OST in the <tt>OST_STATFS</tt> RPC reply. The MDS already checks for this (it sets it internally), similar to the "<tt>OS_STATE_DEGRADED</tt>" flag, but it is an absolute rather than a hint.</p> <p>The code in <tt>osp_pre_update_status()</tt> needs to be cleaned up to avoid <b>clearing</b> the <tt>OS_STATE_NOPRECREATE</tt> flag (and probably <tt>OS_STATE_ENOSPC</tt> and <tt>OS_STATE_ENOINO</tt>, so the OST could send these itself as well). The <tt>obd_statfs</tt> structure is refreshed every few seconds, so the MDS shouldn't need to clear the flags it sets itself.</p> <p>Also, it appears there is a bit of a race in <tt>osp_statfs_interpret()</tt> calling <tt>osp_pre_update_status()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> d-&gt;opd_statfs = *msfs; <span class="code-keyword">if</span> (d-&gt;opd_pre) osp_pre_update_status(d, rc); </pre> </div></div> <p>since there is a window between when the new <tt>&#42;msfs</tt> is stored in <tt>opd_statfs</tt> that is returned by <tt>osp_statfs()</tt>, and when <tt>osp_pre_update_status()</tt> might set the various <tt>OS_STATE_&#42;</tt> flags that the create threads check. That may allow file creation on an OST that should otherwise be unavailable (out of space, disabled, etc). Instead, the new <tt>&#42;msfs</tt> should be passed as an argument to <tt>osp_pre_update_status()</tt> and updated <em>before</em> it is stored into <tt>opd_statfs</tt>.</p> <p>I think the request from <a href="https://jira.whamcloud.com/browse/LU-12036" title="Add option to create new OSTs inactive" class="issue-link" data-issue-key="LU-12036"><del>LU-12036</del></a> could/should probably be integrated here.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35029" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35029</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12025" title="Adding OST may cause EIO - delay activation of new OSTs on existing filesystem" class="issue-link" data-issue-key="LU-12025"><del>LU-12025</del></a> osp: allow OS_STATE_* flags from OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1227336419caed88199f8d21076fb9358070f004</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35029/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35029/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12025" title="Adding OST may cause EIO - delay activation of new OSTs on existing filesystem" class="issue-link" data-issue-key="LU-12025"><del>LU-12025</del></a> osp: allow OS_STATE_* flags from OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9b0ebf78f7919a144673edadc4a95bad84fae2d3</p> <p>This allows the OST to completely disable itself from precreation without having to hack around in the state. Using "degraded" is only partially disabled (can still be used in emergency), and faking "out of space" is a hack.</p> <p>In <a href="https://jira.whamcloud.com/browse/LU-12036" title="Add option to create new OSTs inactive" class="issue-link" data-issue-key="LU-12036"><del>LU-12036</del></a> we should allow mounting the OST with the "<tt>OS_STATE_NOPRECREATE</tt>" flag set, so that it can be mounted but it will not be used.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36872" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36872</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12025" title="Adding OST may cause EIO - delay activation of new OSTs on existing filesystem" class="issue-link" data-issue-key="LU-12025"><del>LU-12025</del></a> osp: allow OS_STATE_* flags from OSTs<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 8f2b85288f973681bd8bd6d8a04f421f57a78a04</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36872/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36872/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12025" title="Adding OST may cause EIO - delay activation of new OSTs on existing filesystem" class="issue-link" data-issue-key="LU-12025"><del>LU-12025</del></a> osp: allow OS_STATE_* flags from OSTs<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: b0194200146a54ee45df208da88dcc6b916fb51f</p> Related LU-12036 LU-17299 LU-12998 Development Rank 1|i00cdr: Rank (Obsolete) 9223372036854775807 Severity