A trivial modification to sanity-flr test_0h shows a serious problem.
If we write to the file before changing the layout flags, the file metadata is corrupted:
But additionally, I'm not sure what the correct behavior is here - should it be possible to remove the prefer flag in this situation, while the other replicas are stale?
And to set stale on the write replica? (That, at least, seems clearly incorrect.)
The effect of this (... if it worked ...) would presumably be that all the replicas are stale, which would be unresolvable, and so should not be allowed. (Separate from the corruption)
There are various implications here.