Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-16374] Implement backup/restore of encrypted files https://jira.whamcloud.com/browse/LU-16374 Lustre <p>Implement backup/restore of encrypted files, according the the HLD as available at <a href="https://datadirectnetworks-my.sharepoint.com/:w:/g/personal/sbuisson_ddn_com/EeWD3Q7Ku69Anntda03QPDUBs6oxRCxtlxELM7xxy-S1qQ" class="external-link" target="_blank" rel="nofollow noopener">https://datadirectnetworks-my.sharepoint.com/:w:/g/personal/sbuisson_ddn_com/EeWD3Q7Ku69Anntda03QPDUBs6oxRCxtlxELM7xxy-S1qQ</a></p> <p>This effort can be divided into the following steps:</p> <ul> <li>security.encdata xattr getting at ldiskfs level</li> <li>security.encdata xattr setting at ldiskfs level</li> <li>security.encdata xattr getting at llite level</li> <li>security.encdata xattr setting at llite level</li> <li>Lustre support for O_TMPFILE open flag</li> <li>lfs fscrypt read</li> <li>lfs fscrypt write</li> <li>tar modifications for backup/restore</li> <li>HSM POSIX copytool modifications for backup/restore</li> </ul> LU-16374

Implement backup/restore of encrypted files

New Feature Minor Open Unresolved Sebastien Buisson Sebastien Buisson Thu, 8 Dec 2022 10:54:19 +0000 Tue, 14 Nov 2023 22:37:51 +0000 Lustre 2.16.0 0 7 <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49410" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49410</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> ldiskfs: round-up enc file size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 93f5269fa71b058ced93bb52c8d276f78cc3b0aa</p> <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49456" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49456</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> ldiskfs: implement security.encdata xattr<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 92434a9ebdc807936524c5ca6bfaf3a01f8bfa87</p> <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49581" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49581</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: align Base64 encoding with RFC 4648 base64url<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a2c20af51ca45120faf91ad277f333f46c32e5a8</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49581/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49581/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: align Base64 encoding with RFC 4648 base64url<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 583ee6911b6cac7f2867a37101cc069b4011b73f</p> <p>Sebastien, can you please add a test case for the bas64 vs. base64url compatibility.</p> <p>Best would be to update <tt>test32_newtarball()</tt> to add a 2.14.0 directory with data-only encryption and a 2.15.0 directory with filename encryption (with base64 name encoding) enabled. The <tt>test_32</tt> script should contain an "ls -l" of the encrypted directories to verify they are all unchanged. This listing would fail for the 2.15.0 filename encrypted directory with a master client due to the base64url encoding, unless <tt>llite.*.filename_enc_use_old_base64=1</tt> is set. </p> <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49945" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49945</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: align Base64 encoding with RFC 4648 base64url<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: d26efca6180f3997d89d9853a3891ee13115d5e8</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49410/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49410/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> ldiskfs: round-up enc file size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ea95b07c6c3869dad6bd925754dec5ea173a3c43</p> <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50561" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50561</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> ldiskfs: implement backup/restore of enc files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b9338f73057f3f11faa8946fccbac657cf52ee8b</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49945/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49945/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: align Base64 encoding with RFC 4648 base64url<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 3a12414a169a6199a5d187ed801aa9577c9f1192</p> <p>I noticed while running <tt>strace tar</tt> that it is always opening files with the equivalent of <tt>O_FILE_ENC</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#define O_FILE_ENC (O_NOCTTY | O_NDELAY) openat(AT_FDCWD, "/mnt/testfs/sparse", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = 3 </pre> </div></div> <p>because of <tt>include/uapi/asm-generic/fcntl.h</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #ifndef O_NDELAY #define O_NDELAY O_NONBLOCK #endif </pre> </div></div> <p>I think this is <em>mostly</em> saved by the additional requirement for <tt>O_DIRECT</tt> (which is <b>not</b> part of <tt>O_FILE_ENC</tt>, but should be). We should consider changing this to use a different flag combination for <tt>O_FILE_ENC</tt> to avoid the risk of accidental issues if <tt>tar</tt> ever starts reading with <tt>O_DIRECT</tt> before we can get the xattr handling implemented.</p> <p>One possibility includes <tt>O_DSYNC</tt>, which doesn't make much sense for <tt>O_RDONLY</tt> files, but would force <b>writes</b> on encrypted restore to be synchronous. With <tt>O_DIRECT</tt> and large enough writes (32MB?) that might be OK, but not ideal for small files. Another possibility is <tt>O_APPEND</tt>, which would prevent multiple writers to an encrypted file during restore, but would otherwise be OK. I haven't looked at any of the options in detail yet, just wanted to point out this issue.</p> <p>"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51640" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51640</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: rename O_FILE_ENC to O_CIPHERTEXT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ecfbc16ad38eb514fb14d1919fdc4f24a96fd403</p> <p>For reference, there was a discussion about fscrypt file backup and restore on the linux-ext4, linux-fscrypt, and linux-fsdevel mailing lists in "<a href="https://lore.kernel.org/linux-ext4/03a87391-1b19-de2d-5c18-581c1d0c47ca@gmail.com/" class="external-link" target="_blank" rel="nofollow noopener">Backup/restore of fscrypt files and directories</a>" and this also referenced other discussions "<a href="https://lore.kernel.org/linux-fscrypt/D1AD7D55-94D6-4C19-96B4-BAD0FD33CF49@dilger.ca/T/#u" class="external-link" target="_blank" rel="nofollow noopener">backup/restore of fscrypt files</a>" and "<a href="https://lore.kernel.org/linux-fsdevel/CAHk-=wh74eFxL0f_HSLUEsD1OQfFNH9ccYVgCXNoV1098VCV6Q@mail.gmail.com" class="external-link" target="_blank" rel="nofollow noopener">fs: interface for directly reading/writing compressed data</a>".</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51640/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51640/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> enc: rename O_FILE_ENC to O_CIPHERTEXT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac522557b1fe3ea2b7275fa6d5df73691b8d06db</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49456/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49456/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16374" title="Implement backup/restore of encrypted files" class="issue-link" data-issue-key="LU-16374">LU-16374</a> ldiskfs: implement security.encdata xattr<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d0a722cb8fb886380e24e8261e8efca09a3262d6</p> Related LU-16259 LU-14677 Development Epic Link EX-5772 Rank 1|i037d3: Rank (Obsolete) 9223372036854775807