Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/30957/
Subject: LU-10286 mdt: deny 2.10 clients to open mirrored files
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 21e39775a0f4f8d7819a49c37b59379a1181f52a

Errno "524" is ENOTSUPP, how about returning EACCES that means 2.10 clients have no permission to access mirrored files.

With patch https://review.whamcloud.com/30957 applied on 2.10.57 client and servers, the test results are:

On 2.10.57 Client2:

[root@onyx-22vm7 tests]# lfs mirror create -N -o 1 -N -o 2 -N -o 3 /mnt/lustre/file1
[root@onyx-22vm7 tests]# stat /mnt/lustre/file1
  File: ‘/mnt/lustre/file1’
  Size: 0               Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d    Inode: 144115205272502273  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-21 07:35:58.000000000 +0000
Modify: 2018-01-21 07:35:58.000000000 +0000
Change: 2018-01-21 07:35:58.000000000 +0000
 Birth: -
[root@onyx-22vm7 tests]# cat /mnt/lustre/file1

Then on 2.10.3 Client1:

[root@onyx-22vm3 ~]# stat /mnt/lustre/file1
  File: ‘/mnt/lustre/file1’
  Size: 0               Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d    Inode: 144115205272502273  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-21 07:35:58.000000000 +0000
Modify: 2018-01-21 07:35:58.000000000 +0000
Change: 2018-01-21 07:35:58.000000000 +0000
 Birth: -
[root@onyx-22vm3 ~]# ls /mnt/lustre/file1 
/mnt/lustre/file1
[root@onyx-22vm3 ~]# cat /mnt/lustre/file1
cat: /mnt/lustre/file1: Unknown error 524

As expected, the 2.10.3 client can't open the mirrored file. However, the error message of "Unknown error 524" is not user-friendly.

This is expected. 2.10 clients will corrupt mirrored files. Please apply patch https://review.whamcloud.com/#/c/30957/1 and I would expect that 2.10 client won't be able to open mirrored files any more.

I set up Lustre filesystem with the following interop configuration on 4 test nodes:

Client1: onyx-22vm3 (2.10.3 RC1)
Client2: onyx-22vm5 (2.10.57)
MDS: onyx-22vm1 (2.10.57)
OSS: onyx-22vm2 (2.10.57)

On 2.10.57 Client2:

[root@onyx-22vm5 tests]# lfs mirror create -N -o 1 -N -o 2 -N -o 3 /mnt/lustre/file1
[root@onyx-22vm5 tests]# stat /mnt/lustre/file1
  File: ‘/mnt/lustre/file1’
  Size: 0               Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d    Inode: 144115205272502273  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-21 05:30:15.000000000 +0000
Modify: 2018-01-21 05:30:15.000000000 +0000
Change: 2018-01-21 05:30:15.000000000 +0000
 Birth: -
[root@onyx-22vm5 tests]# cat /mnt/lustre/file1

Then on 2.10.3 Client1:

[root@onyx-22vm3 ~]# stat /mnt/lustre/file1
  File: ‘/mnt/lustre/file1’
  Size: 0               Blocks: 0          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d    Inode: 144115205272502273  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-01-21 05:30:15.000000000 +0000
Modify: 2018-01-21 05:30:15.000000000 +0000
Change: 2018-01-21 05:30:15.000000000 +0000
 Birth: -
[root@onyx-22vm3 ~]# cat /mnt/lustre/file1

Then on 2.10.57 Client2:

[root@onyx-22vm5 tests]# echo foo > /mnt/lustre/file1
[root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1
[root@onyx-22vm5 tests]# cat /mnt/lustre/file1
foo

Then on 2.10.3 Client1:

[root@onyx-22vm3 ~]# cat /mnt/lustre/file1
foo
[root@onyx-22vm3 ~]# echo goo >> /mnt/lustre/file1
[root@onyx-22vm3 ~]# cat /mnt/lustre/file1
foo
goo

Then on 2.10.57 Client2:

[root@onyx-22vm5 tests]# cat /mnt/lustre/file1
foo
[root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1
lfs mirror resync: '/mnt/lustre/file1' file state error: ro.

2.10.3 Client1 wrote "goo" into the mirrored file /mnt/lustre/file1, but on 2.10.57 Client2, the file data were not updated.

Then on 2.10.57 Client2:

[root@onyx-22vm5 tests]# echo hoo >> /mnt/lustre/file1
[root@onyx-22vm5 tests]# lfs mirror resync /mnt/lustre/file1
[root@onyx-22vm5 tests]# cat /mnt/lustre/file1
foo
goo
hoo

The file data were updated after writing new data and re-syncing.

Then on 2.10.3 Client1:

[root@onyx-22vm3 ~]# cat /mnt/lustre/file1
foo
goo
hoo

The file data were correct on 2.10.3 Client1.

Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: https://review.whamcloud.com/30957
Subject: LU-10286 mdt: deny 2.10 clients to open mirrored files
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0f735407d3582c3b6b4e5c943ecbf57a32d43a73

Now I recall more details. Since 2.10 clients don't verify overlapping extents, they would access mirrored files like normal PFL files, which means they could use any components for I/O. So you're right, we need to define the behavior when mirrored files are being accessed by old clients.

I also looked at the options to return a fake layout to 2.10 clients, but the problem was there are too many places that a layout could be packed and sent to clients. Returning fake layout will have to repair all those code.

For read access, a 2.11 MDS could return a single mirror to 2.10 clients, and if that becomes stale then the MDS would cancel the layout lock and the 2.10 client should get a new layout with the non-STALE mirror?

Yes, I was thinking about the case that read I/O and mirror staling would happen at the same time, so that the read still would return stale data. However, that's probably okay since it would also happen for 2.11 clients.

Do we need an OBD_CONNECT_MIRROR or _FLR flag so the MDS can detect which clients work properly? That is easy to do now, much harder to do later.

Let's add this flag and if a client that don't have this flag is trying to open a mirrored file, let's return an error. This seems the simplest solution. We can come back to make a better solution if necessary.

It's not a question about "whether we should support it", but rather that users will do this whether we tell them to or not. Either it should "work" or there needs to be some mechanism that prevents 2.10 clients from incorrectly accessing these files incorrectly.

For read access, a 2.11 MDS could return a single mirror to 2.10 clients, and if that becomes stale then the MDS would cancel the layout lock and the 2.10 client should get a new layout with the non-STALE mirror?

Similarly, for 2.10 clients opening the file for write would just mark all but one mirror STALE right away. Not the best for performance, but at least correct.

Do we need an OBD_CONNECT_MIRROR or _FLR flag so the MDS can detect which clients work properly? That is easy to do now, much harder to do later.

I thought about this and understood your expectation clearly. Let me explain it a little bit(I did this before but it was on Skype channel).

In your case, there would be a cluster that has mixed 2.11 and 2.10 clients, because obviously mirrored files can only be created by 2.11 clients. If write is supported by 2.10 clients(only writing to the first mirror but not mark the other mirrors stale), then the corresponding files are really messed because reading by different 2.11 clients could return different version of data.

Read support by returning a fake layout would have problem too. After the file has been written by 2.11 clients, the layout cached on 2.10 client would be marked as stale but the 2.10 client has no idea about it, then stale data will be returned from read. Users would think it as a bug.

As you can see, we make huge effort on it but end up with a defective solution. I would rather not support it because only 2.10 clients will be affected(clients prior to 2.10 do not even understand PFL), probably not a big deal.

As described in the original request, testing also needs to be done with 2.10 clients, for both read and write operations. I expect 2.10 clients may be able to read FLR files, but will not write to them properly, possibly writing to the first mirror and not marking the other mirror stale on the MDS.