https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12275 Lustre <p>This ticket is a place-holder to describe work to be done for client-side encryption.</p> <p>The high-level requirements are the following:</p> <ul> <li>encrypt file content</li> <li>encrypt file name</li> <li>have a master key for encryption <ul> <li>per-file encryption key derived from master key</li> <li>file data is no longer accessible after file is deleted</li> </ul> </li> <li>able to change the user key without re-encrypting files</li> <li>deny access to encrypted data when master key is removed from memory on the client</li> <li>work in "batch scheduler" mode</li> </ul> <p>We are proposing to address these requirements by:</p> <ul> <li>conforming to fscrypt kernel API <ul> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/Documentation/filesystems/fscrypt.rst" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/Documentation/filesystems/fscrypt.rst</a></li> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/fs/crypto" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/fs/crypto</a></li> <li>current users are ext4, F2FS, and UBIFS</li> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/include/linux/fscrypt.h" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt.git/tree/include/linux/fscrypt.h</a></li> <li>need to use newly pushed policy v2</li> <li><a href="https://patchwork.kernel.org/patch/10806461/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/10806461/</a></li> <li><a href="https://www.spinics.net/lists/linux-fscrypt/msg01357.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.spinics.net/lists/linux-fscrypt/msg01357.html</a></li> </ul> </li> <li>relying on ext4 encryption principles <ul> <li>file system block size = system page size</li> <li>each filesystem block is encrypted independently in a separate block</li> <li>pages in the page cache always contain clear text data</li> </ul> </li> <li>mutualizing code infrastructure with compression work <ul> <li>same kind of operations, at same code locations</li> </ul> </li> </ul> <p>So the workflow would be the following:</p> <ul> <li>applications see clear text</li> <li>data is encrypted before being sent to servers <ul> <li>then remain untouched</li> </ul> </li> <li>data is decrypted upon receipt from servers <ul> <li>untouched before that</li> </ul> </li> <li>servers only see encrypted data <ul> <li>but do not need to be aware of it</li> </ul> </li> <li>only client nodes have access to encryption keys</li> </ul> <p>Further details will be added as the feature design makes progress.</p> LU-12275

Client-side file data encryption

New Feature Critical Resolved Fixed Sebastien Buisson Sebastien Buisson encryption sec Thu, 9 May 2019 06:37:33 +0000 Tue, 28 Mar 2023 07:12:22 +0000 Sat, 19 Sep 2020 14:54:38 +0000 Lustre 2.14.0 Lustre 2.14.0 0 9 <p>lustre_encryption_threat_model.txt is the description of the threat model for Lustre client-side encryption.<br/> As we want to use kernel's fscrypt library for this feature, fscrypt's threat model is largely valid. This document is inspired from code submitted by Eric Biggers in his fscrypt-key-mgmt-improvements-v6 branch.<br/> <a href="https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6</a></p> <p>lustre_encryption_key_hierarchy.txt is the description of the key hierarchy for Lustre client-side encryption.<br/> As we want to use kernel's fscrypt library for this feature, fscrypt's key hierarchy is largely valid. This document is inspired from code submitted by Eric Biggers in his fscrypt-key-mgmt-improvements-v6 branch.<br/> <a href="https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6</a></p> <p>lustre_encryption_modes_usage.txt is the description of the encryption modes and usage for Lustre client-side encryption.<br/> As we want to use kernel's fscrypt library for this feature, fscrypt's description of encryption modes and usage is largely valid. This document is inspired from code submitted by Eric Biggers in his fscrypt-key-mgmt-improvements-v6 branch.<br/> <a href="https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6</a></p> <p>lustre_encryption_access_semantics.txt is the description of the access semantics for Lustre client-side encryption.<br/> As we want to use kernel's fscrypt library for this feature, fscrypt's description of access semantics is largely valid. This document is inspired from code submitted by Eric Biggers in his fscrypt-key-mgmt-improvements-v6 branch.<br/> <a href="https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/tree/Documentation/filesystems/fscrypt.rst?h=fscrypt-key-mgmt-improvements-v6</a></p> <blockquote> <p>relying on ext4 encryption principles</p> <ul class="alternate" type="square"> <li>file system block size = system page size</li> <li>each filesystem block is encrypted independently in a separate block</li> </ul> </blockquote> <p>It would be better to not tie the encryption directly to the filesystem blocksize, if possible, as this would break the network transparency of the filesystem compared to the backend storage. Also, for ZFS, the blocksize may be up to 1MB and possibly variable per file and per stripe of the file, while the ext4 blocksize is only 4KB. Instead, if we can pick a fixed blocksize for the crypto like 32KB, similar to how the data compression is done. Instead of using the block number as the IV, it should use the byte offset as the IV, since this is agnostic of the blocksize.</p> <p>The drawback is that we may have to write/allocate a larger portion of the files data (rounded up to 32KB or similar), but this would already be true if we are using a feature like <tt>bigalloc</tt> on ext4, or <tt>recordsize</tt> for ZFS.</p> <blockquote> <p>In particular, currently there is no requirement to support unlocking a file<br/> with multiple alternative master keys or to support rotating master keys.</p></blockquote> <p>I thought this was one of the features we were interested in having?</p> <p>Being able to encrypt/decrypt a directory with multiple different keys is definitely something we would like to propose. But if we want to have it directly at the Lustre level, it means we cannot rely on fscrypt.</p> <p>However, basing the Lustre client encryption feature on fscrypt kernel API means we would be able to leverage the fscrypt userspace tool <a href="https://github.com/google/fscrypt" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/google/fscrypt</a>. This tool enables to manipulate encryption policies on directories, and makes use of key wrapping. One of the benefits of this technique is to support using multiple protectors for a policy:<br/> <a href="https://github.com/google/fscrypt#using-multiple-protectors-for-a-policy" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/google/fscrypt#using-multiple-protectors-for-a-policy</a></p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36143" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36143</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: enable client encryption flag<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14558d3d99362697deeee57e02618d5a2dd88e30</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36144" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36144</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: encryption for write path<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: da9ea1a9fd9b29608c9ce3bde8191abd4f74b98f</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36145" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36145</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: decryption for read path<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ddd2bd2e44c7c5e661ef793dc4a78f17c56cae29</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36146" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36146</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: deal with encrypted object size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 16ff787abef944b803f88f9acfd312cfeabf792a</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sebastien" class="user-hover" rel="sebastien">sebastien</a>, what's your plan for tests for these?  I note that none of the patches seem to have tests today.  Are you planning to add a bunch of tests in a separate patch, or tests to each patch, etc?  They're also helpful for review because they show the reviewer the interface(s).</p> <p>Given that all 4 patches are just here to serve one purpose, which is doing IOs (write and read) on encrypted files, I was planning to add a separate patch after them.</p> <p>At the moment, there is no specific interface to exercise this code. You have to use the dummy encryption mode, as described in the mount.lustre man page. A dummy key can be added to the keyring thanks to the following script, after that you just need to mount the client with the 'test_dummy_encryption' option, and all files created under a subdirectory (not the root of the Lustre fs) will be encrypted.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#!/bin/bash mode='\x00\x00\x00\x00' raw="$(printf ""\\\\x%02x"" $(seq 0 63))" if lscpu | grep "Byte Order" | grep -q Little ; then size='\x40\x00\x00\x00' else size='\x00\x00\x00\x40' fi key="${mode}${raw}${size}" echo -n -e "${key}" | keyctl padd logon fscrypt:4242424242424242 @s </pre> </div></div> <p>So it's not possible to apply encryption to the root?</p> <p>Also, that makes sense - thanks.</p> <p>So far I have decided to apply the same restriction as ext4, which forbids encrypting the root of the file system. With ext4, having the root encrypted is a problem for file system check (lost+found cannot be encrypted), but I am not sure it would be a problem for Lustre. However, setting a policy on a directory imposes that this directory is empty. And this is an issue with Lustre's root directory, because it always contains .lustre directory.</p> <p>Maybe we could think about how to release this restriction in the future, but I am not sure this is very limiting. And we still have the possibility to use subdirectory mount if we want clients to see only encrypted stuff.</p> <p>Since the Lustre-visible root directory is already the "<tt>ROOT/</tt>" subdirectory, it would be <em>possible</em> to set the encryption before the <tt>.lustre/</tt> subdirectory is created.</p> <p><b>However</b>, I think this would also cause problems. LFSCK would need to have access to the keyring on the client in order to create files during LFSCK, since it would be against the fscrypt policy to have an unencrypted Lustre <tt>lost+found/</tt> subdirectory where it could generate unencrypted filenames.</p> <p>It probably makes the most sense to keep the root directory unencrypted, and only encrypt subdirectories. If users don't want to expose information about what those subdirectories are, they can use generic names like "<tt>dir1, dir2, dir3</tt>" (maybe matching the UID, since that is visible information anyway).</p> <p>from <a href="https://review.whamcloud.com/36146" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36146</a></p> <blockquote> <p>As far as I understand, ext4 always writes or reads whole blocks, and then uses i_size to present the correct number of bytes to the upper layers. ext4 does not have to do something special regarding file size when encryption is on, because pages are encrypted just before being written to blocks, and decrypted right after being read from blocks.</p></blockquote> <p>Is it possible that the Lustre client/OSS always does the same thing as ext4, reading/writing the full PAGE_SIZE of data, but storing the actual file size on the object(s)? That means some data would be stored explicitly beyond EOF, but would avoid the complexity of storing a separate trusted.enc xattr, which could become out-of-sync with the actual file size in a number of ways.</p> <p>Replying to my own comment:</p> <blockquote> <p>It would be better to not tie the encryption directly to the filesystem blocksize, if possible, as this would break the network transparency of the filesystem compared to the backend storage. Also, for ZFS, the blocksize may be up to 1MB and possibly variable per file and per stripe of the file, while the ext4 blocksize is only 4KB. Instead, if we can pick a fixed blocksize for the crypto like 32KB, similar to how the data compression is done. Instead of using the block number as the IV, it should use the byte offset as the IV, since this is agnostic of the blocksize.</p> <p>The drawback is that we may have to write/allocate a larger portion of the files data (rounded up to 32KB or similar), but this would already be true if we are using a feature like bigalloc on ext4, or recordsize for ZFS.</p></blockquote> <p>One possible option would be to do the encryption in chunks of the minimum <tt>PAGE_SIZE</tt>=4KiB, and use the IV for each chunk based on the 4KB index and not the <em>actual</em> <tt>PAGE_SIZE</tt> if it is different. This means that for the common <tt>PAGE_SIZE</tt>=4KiB the encryption would work exactly the same (i.e. compatible with files written by x86 clients), but for <tt>PAGE_SIZE</tt>=64KiB there would be 16 4KiB chunks within the page that are encrypted/decrypted separately.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36191" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36191</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> tests: exercise file content encryption/decryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d3ee36219e28a8ca7d4e5e9782eb9a21c19af106</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36238" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36238</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> osd: make osd layer always send full pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 26cdc0abb9d74d92a86c6fcb28efc78d97b39517</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36360" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36360</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: reserve flags for client side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4cc6598e2cd2959efb86b0c741b7a518217254e0</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36361" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36361</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: deal with encrypted object size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bbeb3c24bf69dae3e9b1c2fbf84061e8ba191cc5</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36433" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36433</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: control client-side 'encrypt' mount option<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2c3d2f44ad1e4d5ea8cd0cc5434caff9281fde96</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36360/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36360/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: reserve flags for client side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4f9632f9701130afc245810dde54035ab7caf2d3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36238/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36238/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> osd: make osd layer always send complete pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7b136af8f46024fd48a773e5a79b817bca05d9e8</p> <p>Question for Sebastien and/or Andreas:</p> <p>Would this be incompatible with allowing unaligned direct i/o?  It seems clear the answer is yes - Today, buffered i/o can easily be forced to work in full pages (which this code does) and direct i/o <b>must</b> work in full pages due to alignment requirements.</p> <p>I had some interest a while back in lifting the alignment requirement for DIO, which could improve performance for some scenarios of interest in certain cases.</p> <p>So it seems it would be necessary to prevent unaligned DIO on encrypted files, which would be relatively straightforward, if potentially a little confusing.</p> <p>Looking at <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> osd: make osd layer always send complete pages / <a href="https://review.whamcloud.com/36238/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36238/</a> it doesn't seem like it would cause trouble, in that it uses the lnb_len provided, so it doesn't so much force sending complete pages as it forces filling the client buffer, which I think doesn't have to be full pages...</p> <p>Actually, it occurs to me that we could fall back to buffered i/o for DIO on encrypted files, if we decide that's important.  It's unusual, but it's something that GPFS already does, so it's not unprecedented.</p> <p>Any strong reactions to this or aspects I've missed in the context of this work?</p> <p>Hi Patrick,</p> <p>As explained in the attachment <a href="https://jira.whamcloud.com/secure/attachment/32658/lustre_encryption_access_semantics.txt" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/32658/lustre_encryption_access_semantics.txt</a>, "direct I/O is not supported on encrypted files. Attempts to use direct I/O on such files will fall back to buffered I/O."</p> <p>So yes, just like GPFS <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Ah, OK!  Well that's easy, then.  Thanks.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37672" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37672</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: avoid encrypted files corruption with truncate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a46c9b790b338109b2aa685287f88cf39d951b62</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37673" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37673</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: ioctls to handle encryption policies<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 304c015b3778759ed1f6af5f229ece5ce5ccc4c7</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37794" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37794</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: attempt to support truncate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 10c7a8979fb3f1230ac254ec4a9a45f51ca4afaa</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38127" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38127</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: add llcrypt as file encryption library<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 609fe7b1c72b2d60367ae128889925a879a017a1</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38430" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38430</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: atomicity of encryption context getting/setting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5238d2ac56268a200e5ed1df89679964cd52735b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38127/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38127/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: add llcrypt as file encryption library<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a813e81870096bcfecbe12aeeed8e1b0114cd474</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38702" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38702</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: encryption support for DoM files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2d9d85e86a77bfb09ad96e9dd7c2ac6add711f30</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38759" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38759</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: documentation for client-side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 63ff8c506a0a7941e03545519e9242078fa198c4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38759/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38759/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: documentation for client-side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f2bf564e691ea8f64843f5c77ae1a8bd60f1b70b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36143/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36143/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: enable client side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 28be31137cd22223113e461f74098c92ba6d71e4</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38881" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38881</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: introduce null alg for client side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0f5daebd7911e8c4568d255b6ec14e7627bc116c</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38882" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38882</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: force file name encryption policy to null<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cf114541eda591fc2cf66ae45b0b85fec37b4588</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36433/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36433/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: control client side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3042bcd709ebfea4cf543eb6e8aca330a6cafe9f</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38918" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38918</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: check if page is empty with ZERO_PAGE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1f0a7547c7e8e8d6984ed591cac3671fe51179ac</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36144/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36144/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: encryption for write path<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a9ed5b149646f91f23839bbe2d755542d129f5b7</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36145/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36145/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: decryption for read path<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eecf86131d099242d2e8c1f5d6be241ec1416c9a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36146/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36146/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: deal with encrypted object size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 83d660436a164758fd4a29c1433d11c0f4591196</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37794/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37794/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: support truncate for encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: adf46db962f657b74bd38db27e7b320aaee3cdd5</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38967" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38967</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: O_DIRECT for encrypted file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a42ab045b26bc52f0ae13cac929c1476ab38224</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36191/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36191/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> tests: exercise file content encryption/decryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 598c48707cffbb813058f74bb1e663bcdde3c80e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37673/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37673/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: ioctls to handle encryption policies<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3973cf8dc955c773a5f9da13216252644aa3949f</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39220" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39220</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: restrict fallocate on encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9725d412c9af1996342afbab5b0dc8e19a176ddb</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39315" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39315</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> tests: encryption with different client PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f9cc5e3a68fa597b4ee368f7dd3b71152c8ad2c5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38881/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38881/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: introduce null algo for filename encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c60b7d9f571748fb055d29cd019709f9e965a84d</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38882/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38882/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: force file name encryption policy to null<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 11fcbfa9de4a5170abc2c5df2a6e4e02f0f84268</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38430/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38430/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: atomicity of encryption context getting/setting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 40d91eafe257fb407d27c54cd2f7ae9961672f60</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38702/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38702/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: encryption support for DoM files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a71586d4ee8d6f039a413e2a0fd791db847a3c19</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38918/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38918/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: check if page is empty with ZERO_PAGE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 45c26bc11b7b1ec6b6b4773fb9a40e0a655f7d33</p> <p>Neil Brown (neilb@suse.de) uploaded a new patch: <a href="https://review.whamcloud.com/39459" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39459</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: use memchr_inv() to check if page is zero.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5a36eee3767f27e0bf5065c2dc1785e6c2d7aa5a</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39558" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39558</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: ldiskfs not aware of client-side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c24b7479f476d371cc6b8c2e852a8579e3808a4b</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39617</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: verify dir is empty when setting enc policy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9cbe9dfd7da0fc57c643eea2d95ebf4d3fb98491</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39459/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39459/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: use memchr_inv() to check if page is zero.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: afee2380c105c37e440aaa9ec588cd27189bc18e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38967/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38967/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: O_DIRECT for encrypted file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 728036f25635a9b14310eded1761cf6cd0bacb1a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39220/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39220/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: restrict fallocate on encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a7870fb9568bf753d50eeead71a59dfe07db1d20</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39558/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39558/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: ldiskfs not aware of client-side encryption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ad444ed9836320c6ae8b770ff96edd6b0fe4f0d4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39315/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39315/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: encryption with different client PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac5fcdce025b4825500c0308d89dfdab1faece51</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39617/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39617/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: verify dir is empty when setting enc policy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e8f74fb0f5c9306ee5a099133799e03e09ca8e47</p> <p>I believe that everything planned for 2.14 has now landed</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49828" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49828</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> tests: skip new nodemap params on old MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f69c3cd23c4d6e128d95de781f1db5cfe68dec6f</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49828/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49828/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> tests: skip new nodemap params on old MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 93230059abe9dfe39a8b72cb8fc31bab1cadc7b6</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50023" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50023</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: disable bio functions on client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0ebb7e37b85f8e881985bdcb45d3e16ace37c1f0</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50140" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50140</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: disable bio functions on client<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 546cb30c00fb1db1af59cb88396e84e362370dcf</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50023/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50023/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: remove bio functions in fscrypt compat<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d328818a456daf30c20c8df0aa0be9dd2a2b6a9e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50140/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50140/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: remove bio functions in fscrypt compat<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 6ce5b0bc881389003e90d1201d468bc099251ada</p> Related LU-16091 LUDOC-477 LU-15003 LU-14677 LU-7371 LU-14045 LU-14149 LU-16085 LU-16091 LU-13717 Development Rank 1|i00fzz: Rank (Obsolete) 9223372036854775807