[LU-12738] PFL: append of PFL file should not instantiate full layout

Related — Mon, 8 Jan 2024 20:13:13 +0000

This is a continuation of ~~LU-9341~~, created to track related work which has not been completed yet.
-------

Appending to a PFL file will cause all layout components to be instantiated because it isn't possible to know what the ending offset is at the time the write is started.

It would be better to avoid this, potentially by locking/instantiating some large(r), but not gigantic range beyond current EOF, and if that fails retry the layout intent? The client must currently be in charge of locking the file during append, so it should know at write time how much of the file to instantiate, and it could retry.

-------

A detailed breakdown of the remaining issue & a possible approach to fixing it (originally from comment-252605) :

There are two basic problems/constraints we are trying to meet.

Append writes must be atomic, meaning two things:
1. No "write tearing". Every byte of a particular O_APPEND write must be adjacent, no gaps. (This is true of any write, but it is more relevant for O_APPEND writes, which always start at EOF, so if we are not careful, a restarted append write will restart at a new EOF.)

2. They must not be mixed with other O_APPEND writes. If two O_APPEND writes, A and B, are racing, either AB or BA is valid, but it is not valid for any part of A or B to overwrite the other one. This is of course not true of regular writes, which are started at a specific offset. This means that if a regular write is racing with an O_APPEND write, it can overwrite part of the O_APPEND write. This is acceptable.

The first problem (write tearing) is solved by getting the file size before starting the write and using the same one throughout the O_APPEND write. (The current code checks the file size repeatedly, I believe for every iteration of cl_io_loop().) This means that if another write races with our O_APPEND write we must not 'tear' the O_APPEND write by moving to a new EOF in the middle of the write. Note that we must also retain the size across I/O restarts, for the case where we have to update the layout or resend RPCs in the middle of an O_APPEND write.

The second problem requires that we allow only one O_APPEND write at a time. There are probably a few ways to solve this, but I think the correct way (it is definitely the simplest way) is to add another bit to the MDT IBITS lock, an MDS_INODELOCK_APPEND bit. All O_APPEND writes must ask for this bit in PW mode before starting to write. (We cannot use the MDS_INODELOCK_LAYOUT bit for this exclusion because the server revokes our LAYOUT lock bit when we have to update the file layout.) The APPEND lock must be held across i/o restarts, so it should be taken before the LAYOUT lock. (Or it could possibly be taken with the LAYOUT lock bit? We have to be careful about ordering/pairing issues with the LAYOUT and APPEND bits, I have not thought about this carefully yet.)

Note that excluding O_APPEND writes does require excluding multiple O_APPEND writes on the same node as well. This can be done using the local tree_lock in the write path, locking it from 0 to EOF.

This combination of things should allow not instantiating the full layout and locking every object. It's a fair bit of work.

Note of course that this is a split client/server solution, so it will need a compatibility flag so the client knows it can use the O_APPEND flag. The good news is that this should interop safely with older clients - The older clients simply instantiate and lock everything for O_APPEND, which will give the correct exclusion vs newer clients.

Whamcloud Community JIRA

[LU-12738] PFL: append of PFL file should not instantiate full layout