https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6218 Lustre <p>A site had two last_rcvd files corrupted on two OSTs. They were able to truncate the files and the OSTs mounted OK. But I wonder whether we could increase data redundancy for meta data such as the last_rcvd file, to make it harder to corrupt in the first place (or more accurately to make it easier for scrub to repair it should it ever get corrupted).</p> <p>The OIs already get two copies of its data blocks as they are ZAPs. But other meta data like last_rcvd get only one copy of the data. The copies property can only be applied at per file system granularity. We can put those files under a separate dataset, e.g. lustre-ost1/ost1/META, and set copies=2 for it. But it'd complicate the code as now there's two datasets per OST.</p> LU-6218

osd-zfs: increase redundancy for OST meta data

Improvement Minor Resolved Fixed Isaac Huang Isaac Huang zfs Fri, 6 Feb 2015 06:32:35 +0000 Sun, 31 Jul 2016 13:52:48 +0000 Thu, 20 Aug 2015 13:18:03 +0000 Lustre 2.8.0 0 6 <p>Since the OST has direct access to the DMU code, is it possible to somehow flag the dnode at create or write time to generate a ditto block copy only on that inode? I've long thought that this is useful for last_rcvd, OI files, config files, etc. just writing two copies of these files at the OSD level wouldn't actually solve this problem because ZFS wouldn't know they are copies, and couldn't automatically repair one on error. </p> <p>I just looked at DMU code and it seemed like all we'd need to do is to create these objects with type <em>DMU_OTN_UINT8_METADATA</em>, instead of <em>DMU_OT_PLAIN_FILE_CONTENTS</em> (see <em>dmu_write_policy()</em>).</p> <p>At osd-zfs, can I identify such objects by using conditional <em>"fid_seq(fid) == FID_SEQ_LOCAL_FILE"</em>?</p> <p>Would this make these objects inaccessible if mounted directly via ZPL? That would make last_rcvd, LAST_ID, CATALOGS, other llog files, etc inaccessible for local mounts, which breaks one of the important ZFS compatibility features that we've kept to be able to mount the filesystem. I didn't see any obvious checks for this in the zpl_read() or zpl_write() code paths, but I do see such a check in <tt>zpl_unlinked_drain()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ASSERT((doi.doi_type == DMU_OT_PLAIN_FILE_CONTENTS) || (doi.doi_type == DMU_OT_DIRECTORY_CONTENTS)); </pre> </div></div> <p>but I don't think we will encounter this in practice since Lustre rarely deletes internal file objects except llog files.</p> <p>The DMU_OTN_UINT8_METADATA type definitely shouldn't be used for FID_SEQ_LOCAL ZAPs, at most it should only be used for regular files.</p> <p>Looks like we'd hit that assertion only if:</p> <ol> <li>A DMU_OTN_UINT8_METADATA object is removed, either by ZPL or osd-zfs (with the upcoming <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a> fix - and I can just directly free such objects in that patch to eliminate this possibility).</li> <li>Before it's actually freed the system crashed (or ZPL forced umount), so the object stayed in the ZPL delete queue.</li> <li>The dataset is mounted by ZPL (not read-only)</li> </ol> <p>I'll experiment a bit with a simple patch.</p> <p>Isaac Huang (he.huang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13741</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6218" title="osd-zfs: increase redundancy for OST meta data" class="issue-link" data-issue-key="LU-6218"><del>LU-6218</del></a> osd-zfs: more ditto copies<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 267c12243093e1fd2c92f222a6bac0167986483b</p> <p>Everything looked fine. The files showed up in the ZPL name space and I was able to r/w them. And <em>zdb</em> showed 2 copies of data blocks:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-44vm1 ost1]# ls -li last_rcvd 143 -rw-r--r-- 1 root root 8448 Dec 31 1969 last_rcvd [root@eagle-44vm1 ost1]# zdb -e -dddddd lustre-ost1/ost1 143 ...... Object lvl iblk dblk dsize lsize %full type 143 1 16K 128K 9.00K 128K 100.00 uint8 (K=inherit) (Z=inherit) Indirect blocks: 0 L0 0:25f93600:1200 0:3005dcc00:1200 20000L/1200P F=1 B=646/646 </pre> </div></div> <p>Then I removed last_rcvd, umount, and mount again - didn't hit the assertion in zfs_unlinked_drain(). So it was removed from the delete queue and freed before umount. I also tested <em>zfs send/recv</em> and it worked fine.</p> <p>Great.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> Do you happen to know what size the fs_log_size() in test-framewrok.sh returns? I'm wondering whether I should double the size returned for osd-zfs, but couldn't figure out what size fs_log_size() was actually returning.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13741/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13741/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6218" title="osd-zfs: increase redundancy for OST meta data" class="issue-link" data-issue-key="LU-6218"><del>LU-6218</del></a> osd-zfs: increase redundancy for meta data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d9e86108724c06e3e6d25081caaf5803abf4416c</p> <p>Are there any performance implications from this change? Performance is already a problem on MDTs. This redundancy applies there as well, yes? Is the impact reasonable enough to make this the default there?</p> <p>Isaac</p> <p>Are you able to answer Chris's question about performance?</p> <p>Peter</p> <p>The patch added one additional copy for data blocks of a small number of small files, e.g. last_rcvd. The added overhead is trivial compared to the OIs which already get an additional copy.</p> Related Development Rank 1|hzx5t3: Rank (Obsolete) 17393