https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10286 Lustre <p>This task is to verify the client should be running okay when mirrored files are being accessed by old version of Lustre. There should be no system crash or any other problems that stops old file system from accessing plain and PFL files. However, it's acceptable for old clients to see IO errors when it is trying to access mirrored files.</p> <p>Andreas once mentioned that when a mirrored file is being accessed by an old client, the MDS should be able to make a fake PFL layout from one of mirror so that old clients can still read the data.</p> LU-10286

Verify the behaviors when mirrored files are being accessed by old clients

Technical task LU-9771 Blocker Resolved Fixed Sarah Liu Jinshan Xiong FLR Mon, 27 Nov 2017 20:20:38 +0000 Fri, 9 Feb 2018 06:01:40 +0000 Fri, 9 Feb 2018 06:01:40 +0000 Lustre 2.11.0 Lustre 2.11.0 0 6 <p>There are a few different cases that are of interest here:</p> <ul class="alternate" type="square"> <li>2.9 or earlier clients that do not understand PFL at all. I expect they will get EIO or similar error when trying to access an FLR file, since they don't understand composite files</li> <li>2.10 clients + 2.11 MDS. The client understand composite layouts, but not FLR, and the MDS is FLR-aware. If a non-FLR client opens the file for write, it could mark all but one mirror as STALE, and allow the client to access the stale component. Alternately, it could deny such clients write access, but allow it read access.</li> <li>2.10 clients + 2.10 MDS. Both client and MDS understand composite layouts, but not FLR. What happens if the client tries to read the file? Success or error (both are OK, though it would be better if such a client could at least read an FLR file)? What happens if the client writes the file? If this is allowed, it would result in the mirrors becoming out of sync, but not marked STALE.</li> </ul> <p>I have a system configured as 2.11 servers, one 2.11 client and one 2.9.0 client<br/> 1. on the 2.11 client, create 1 pfl file, 1 flr file with plain layout, and 1 flr file with composite layout<br/> 2. on the 2.9 client, got these messages when try to access these files and when I do "ls -al" :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-77 lustre]# ls foo-ext foo-flr foo-pfl foo-plain-2.9 [root@onyx-77 lustre]# ls -al [329391.090438] LustreError: 57728:0:(lov_internal.h:100:lsm_op_find()) unrecognized lsm_magic 0bd60bd0 [329391.102999] LustreError: 57728:0:(lov_internal.h:100:lsm_op_find()) Skipped 3 previous similar messages [329391.115668] LustreError: 57728:0:(lov_pack.c:213:lov_verify_lmm()) bad disk LOV MAGIC: 0x0BD60BD0; dumping LMM (size=552): [329391.130044] LustreError: 57728:0:(lov_pack.c:213:lov_verify_lmm()) Skipped 3 previous similar messages [329391.142376] LustreError: 57728:0:(lov_pack.c:222:lov_verify_lmm()) FF0BFF0B2802000003000000010005000200000000000000000000000000000001000100100000000000000000000000FFFFFFFFFFFFFFFF10010000380000000000000000000000000000000000000001000200100000000000000000000000000010000000000048010000380000000000000000000000000000000000000002000200000000000000100000000000FFFFFFFFFFFFFFFFFF0100003800000000000000000000000000000000000000010003001000000000000000000000000000100000000000FF010000380000000000000000000000000000000000000002000300000000000000100000000000FFFFFFFFFFFFFFFFFF0100003800000000000000000000000000000000000000FF0BFF0B01000000030000000000000001040000020000000000100001000000040000000000000000000000000000000000000000000000FF0BFF0B01000000030000000000000001040000020000000000100001000000040000000000000000000000000000000000000001000000FF0BFF0B0100000003000000000000000104000002000000000010000200FFFF0000000000000000000000000000000000000000FFFFFFFFFF0BFF0B0100000003000000000000000104000002000000000010[329391.251564] LustreError: 57728:0:(lov_pack.c:222:lov_verify_lmm()) Skipped 3 previous similar messages [329391.266288] LustreError: 57728:0:(lcommon_cl.c:181:cl_file_inode_init()) Failure to initialize cl object [0x200000401:0x3:0x0]: -22 [329391.283577] LustreError: 57728:0:(lcommon_cl.c:181:cl_file_inode_init()) Skipped 3 previous similar messages [329391.296622] LustreError: 57728:0:(llite_lib.c:2300:ll_prep_inode()) new_inode -fatal: rc -22 [329391.307933] LustreError: 57728:0:(llite_lib.c:2300:ll_prep_inode()) Skipped 1 previous similar message ls: cannot access foo-ext: Invalid argument ls: cannot access foo-pfl: Invalid argument ls: cannot access foo-flr: Invalid argument total 8 drwxr-xr-x 3 root root 4096 Dec 22 15:56 . drwxr-xr-x. 3 root root 4096 Dec 18 20:52 .. -?????????? ? ? ? ? ? foo-ext -?????????? ? ? ? ? ? foo-flr -?????????? ? ? ? ? ? foo-pfl -rw-r--r-- 1 root root 0 Dec 22 15:56 foo-plain-2.9 [root@onyx-77 lustre]# </pre> </div></div> <p>It probably makes sense to improve these error messages to consolidate them to at most one message per unknown magic, or similar. It probably isn't useful to dump the long hex string to the console.</p> <p>I have captured <a href="https://jira.whamcloud.com/browse/LU-10535" title="Improve error message handling when mirrored files are accessed by older clients" class="issue-link" data-issue-key="LU-10535"><del>LU-10535</del></a> to address the error handling improvement so that we can resolve this ticket as the testing is complete and work the improvement separately.</p> <p>As described in the original request, testing also needs to be done with 2.10 clients, for both read and write operations. I expect 2.10 clients may be able to read FLR files, but will not write to them properly, possibly writing to the first mirror and not marking the other mirror stale on the MDS. </p> <p>I thought about this and understood your expectation clearly. Let me explain it a little bit(I did this before but it was on Skype channel).</p> <p>In your case, there would be a cluster that has mixed 2.11 and 2.10 clients, because obviously mirrored files can only be created by 2.11 clients. If write is supported by 2.10 clients(only writing to the first mirror but not mark the other mirrors stale), then the corresponding files are really messed because reading by different 2.11 clients could return different version of data.</p> <p>Read support by returning a fake layout would have problem too. After the file has been written by 2.11 clients, the layout cached on 2.10 client would be marked as stale but the 2.10 client has no idea about it, then stale data will be returned from read. Users would think it as a bug.</p> <p>As you can see, we make huge effort on it but end up with a defective solution. I would rather not support it because only 2.10 clients will be affected(clients prior to 2.10 do not even understand PFL), probably not a big deal.</p> <p>It's not a question about "whether we should support it", but rather that users will do this whether we tell them to or not. Either it should "work" or there needs to be some mechanism that prevents 2.10 clients from incorrectly accessing these files incorrectly. </p> <p>For read access, a 2.11 MDS could return a single mirror to 2.10 clients, and if that becomes stale then the MDS would cancel the layout lock and the 2.10 client should get a new layout with the non-STALE mirror?</p> <p>Similarly, for 2.10 clients opening the file for write would just mark all but one mirror STALE right away. Not the best for performance, but at least correct. </p> <p>Do we need an OBD_CONNECT_MIRROR or _FLR flag so the MDS can detect which clients work properly? That is easy to do now, much harder to do later. </p> <p>Now I recall more details. Since 2.10 clients don't verify overlapping extents, they would access mirrored files like normal PFL files, which means they could use any components for I/O. So you're right, we need to define the behavior when mirrored files are being accessed by old clients.</p> <p>I also looked at the options to return a fake layout to 2.10 clients, but the problem was there are too many places that a layout could be packed and sent to clients. Returning fake layout will have to repair all those code.</p> <blockquote> <p>For read access, a 2.11 MDS could return a single mirror to 2.10 clients, and if that becomes stale then the MDS would cancel the layout lock and the 2.10 client should get a new layout with the non-STALE mirror?</p></blockquote> <p>Yes, I was thinking about the case that read I/O and mirror staling would happen at the same time, so that the read still would return stale data. However, that's probably okay since it would also happen for 2.11 clients.</p> <blockquote> <p>Do we need an OBD_CONNECT_MIRROR or _FLR flag so the MDS can detect which clients work properly? That is easy to do now, much harder to do later.</p></blockquote> <p>Let's add this flag and if a client that don't have this flag is trying to open a mirrored file, let's return an error. This seems the simplest solution. We can come back to make a better solution if necessary.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30957" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30957</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10286" title="Verify the behaviors when mirrored files are being accessed by old clients" class="issue-link" data-issue-key="LU-10286"><del>LU-10286</del></a> mdt: deny 2.10 clients to open mirrored files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0f735407d3582c3b6b4e5c943ecbf57a32d43a73</p> <p>I set up Lustre filesystem with the following interop configuration on 4 test nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Client1: onyx-22vm3 (2.10.3 RC1) Client2: onyx-22vm5 (2.10.57) MDS: onyx-22vm1 (2.10.57) OSS: onyx-22vm2 (2.10.57) </pre> </div></div> <p>On 2.10.57 Client2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm5 tests]# lfs mirror create -N -o 1 -N -o 2 -N -o 3 /mnt/lustre/file1 [root@onyx-22vm5 tests]# stat /mnt/lustre/file1 File: ‘/mnt/lustre/file1’ Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-21 05:30:15.000000000 +0000 Modify: 2018-01-21 05:30:15.000000000 +0000 Change: 2018-01-21 05:30:15.000000000 +0000 Birth: - [root@onyx-22vm5 tests]# cat /mnt/lustre/file1 </pre> </div></div> <p>Then on 2.10.3 Client1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm3 ~]# stat /mnt/lustre/file1 File: ‘/mnt/lustre/file1’ Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-21 05:30:15.000000000 +0000 Modify: 2018-01-21 05:30:15.000000000 +0000 Change: 2018-01-21 05:30:15.000000000 +0000 Birth: - [root@onyx-22vm3 ~]# cat /mnt/lustre/file1 </pre> </div></div> <p>Then on 2.10.57 Client2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm5 tests]# echo foo &gt; /mnt/lustre/file1 [root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1 [root@onyx-22vm5 tests]# cat /mnt/lustre/file1 foo </pre> </div></div> <p>Then on 2.10.3 Client1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm3 ~]# cat /mnt/lustre/file1 foo [root@onyx-22vm3 ~]# echo goo &gt;&gt; /mnt/lustre/file1 [root@onyx-22vm3 ~]# cat /mnt/lustre/file1 foo goo </pre> </div></div> <p>Then on 2.10.57 Client2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm5 tests]# cat /mnt/lustre/file1 foo [root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1 lfs mirror resync: '/mnt/lustre/file1' file state error: ro. </pre> </div></div> <p>2.10.3 Client1 wrote "goo" into the mirrored file /mnt/lustre/file1, but on 2.10.57 Client2, the file data were not updated.</p> <p>Then on 2.10.57 Client2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm5 tests]# echo hoo &gt;&gt; /mnt/lustre/file1 [root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1 [root@onyx-22vm5 tests]# cat /mnt/lustre/file1 foo goo hoo </pre> </div></div> <p>The file data were updated after writing new data and re-syncing.</p> <p>Then on 2.10.3 Client1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm3 ~]# cat /mnt/lustre/file1 foo goo hoo </pre> </div></div> <p>The file data were correct on 2.10.3 Client1.</p> <p>This is expected. 2.10 clients will corrupt mirrored files. Please apply patch <a href="https://review.whamcloud.com/#/c/30957/1" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30957/1</a> and I would expect that 2.10 client won't be able to open mirrored files any more.</p> <p>With patch <a href="https://review.whamcloud.com/30957" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30957</a> applied on 2.10.57 client and servers, the test results are:</p> <p>On 2.10.57 Client2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm7 tests]# lfs mirror create -N -o 1 -N -o 2 -N -o 3 /mnt/lustre/file1 [root@onyx-22vm7 tests]# stat /mnt/lustre/file1 File: ‘/mnt/lustre/file1’ Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-21 07:35:58.000000000 +0000 Modify: 2018-01-21 07:35:58.000000000 +0000 Change: 2018-01-21 07:35:58.000000000 +0000 Birth: - [root@onyx-22vm7 tests]# cat /mnt/lustre/file1 </pre> </div></div> <p>Then on 2.10.3 Client1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm3 ~]# stat /mnt/lustre/file1 File: ‘/mnt/lustre/file1’ Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-01-21 07:35:58.000000000 +0000 Modify: 2018-01-21 07:35:58.000000000 +0000 Change: 2018-01-21 07:35:58.000000000 +0000 Birth: - [root@onyx-22vm3 ~]# ls /mnt/lustre/file1 /mnt/lustre/file1 [root@onyx-22vm3 ~]# cat /mnt/lustre/file1 cat: /mnt/lustre/file1: Unknown error 524 </pre> </div></div> <p>As expected, the 2.10.3 client can't open the mirrored file. However, the error message of "Unknown error 524" is not user-friendly.</p> <p>Errno "524" is <tt>ENOTSUPP</tt>, how about returning <tt>EACCES</tt> that means 2.10 clients have no permission to access mirrored files.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30957/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30957/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10286" title="Verify the behaviors when mirrored files are being accessed by old clients" class="issue-link" data-issue-key="LU-10286"><del>LU-10286</del></a> mdt: deny 2.10 clients to open mirrored files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 21e39775a0f4f8d7819a49c37b59379a1181f52a</p> Related LU-10535 Development Rank 1|hzzoaf: Rank (Obsolete) 9223372036854775807