Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.10.0
-
3
-
9223372036854775807
Description
Appending to a PFL file will cause all layout components to be instantiated because it isn't possible to know what the ending offset is at the time the write is started.
It would be better to avoid this, potentially by locking/instantiating some large(r), but not gigantic range beyond current EOF, and if that fails retry the layout intent? The client must currently be in charge of locking the file during append, so it should know at write time how much of the file to instantiate, and it could retry.
Attachments
Issue Links
- is duplicated by
-
LU-10665
DoM: append to file causes OST component initialization
-
- Resolved
-
- is related to
-
-
- Open
-
-
LU-10176
Data-on-MDT phase II
-
- Open
-
-
-
- Open
-
-
-
- Open
-
-
-
- Resolved
-
-
LU-12738 PFL: append of PFL file should not instantiate full layout
-
- Open
-
-
-
- Open
-
- is related to
-
-
- Open
-
-
LU-8998 Progressive File Layout (PFL)
-
- Resolved
-
Activity
Rather than a boolean for append_onestripe_enable, how about "create_append_stripe_count=N" where N=0 is to disable and use the filesystem default? That would allow admins to set whatever count fits best with the general workload. It would also fit better with my other concern about controlling what pool that (single) stripe is on, so have "create_append_stripe_pool=S". (Otherwise it would use any OST, and a large log file single-striped could fill up an expensive SSD target. Or if the admin knows all log files are small, just go ahead and keep them exclusively on SSDs for small write performance!) And I guess "create_append_stripe_width=N" for completeness.
My goal here is to keep the interface as simple as possible so that it can be implemented, tested, landed, backported, and deployed in a reasonable time, since the more complex it is the more chance of bugs, longer testing, more conflicts/dependencies, etc.
I guess I would be OK with append_layout_enable=<stripe_count> instead of only 0, 1 (0 has the added benefit of meaning "use the default stripe count" anyway), though there are performance optimizations that can be done for one-stripe files (e.g. patch https://review.whamcloud.com/31553 "LU-10782 llite: simple layout append tiny write") that would be lost if the files are striped across OSTs. I guess append_layout_pool=<pool> wouldn't be too much more complex. I'd rather avoid the full spectrum of "create complex file layout via tunable parameters" for this issue. I think the stripe_width is so rarely used, and is irrelevant for 1-stripe files, that it doesn't warrant a separate parameter - we can use the filesystem default stripe_width for this.
If we want to allow a complex file layout at some point in the future, this could be done with append_layout_enable=[FID] and LU-9982, but I don't think that is needed for most cases. We could also leverage patch https://review.whamcloud.com/33126 "LU-11234 lod: add pool selection layout_policy" to specify a different pool for specific file extensions like *.log *.out, etc., and that could also integrate with LU-9982 to specify a complex layout, but neither of those patches are themselves finished, they don't necessarily catch all log files (which overwhelmingly use O_APPEND), nor are they necessarily simple enough to make a trivial backport to 2.10.
Nathan, did you ever check/post the file size distribution for O_APPEND log files at your site? I see comment-245248 from Stephane, but nothing from you here or in the other DDN tickets in Jira. If all the log files are all small (Stephane reported 92% smaller than 13KB and 100% smaller than 128KB), then the OST they are located on doesn't really matter, and storing them on flash might make sense anyway because that is faster and applications will eventually be blocked on their stdout/stderr if it is taking a long time. If there is an upper limit on the size of such log files, that would avoid filling flash OSTs/MDTs, and be a generally useful feature as well (IMHO at least).
Unfortunately, choosing how many to instantiate isn't an option - If it were easy to do that, we probably wouldn't be having this conversation. Not instantiating the whole layout for appends is quite challenging for subtle reasons around eliminating the possibility of write-splitting in the scenario where more than one client is writing at once and some writes span the boundary between instantiated and uninstantiated components. The only ways we came up with of doing that which are sure to work are fairly heavy handed. (The details and things we tried are buried in the ~40 previous comments on this bug.)
So Andreas suggested we sidestep this by creating a special layout, as almost all O_APPEND users probably prefer this anyway, and this is straightforward to implement.
"as the lower write latency could improve application performance if they are blocked on the log writes"
I don't think in general that DOM has lower write latency? It's got massively better latency for the first write to a newly created file, but after the first write, don't we expect the latency to be the same? Even if the MDT is flash and the OST is not, I think we expect it to go in cache anyway, right?
But, then again, I suppose APPEND is being used because multiple threads might be writing to this file, and in that case, we're going to spam syncs from lock contention, so it would be very good to be on flash rather than spinning disk. Hmm.
It seems like a "nice to have", but it does seem useful.
Andreas, Mike,
Rather than a boolean for append_onestripe_enable, how about "create_append_stripe_count=N" where N=0 is to disable and use the filesystem default? That would allow admins to set whatever count fits best with the general workload. It would also fit better with my other concern about controlling what pool that (single) stripe is on, so have "create_append_stripe_pool=S". (Otherwise it would use any OST, and a large log file single-striped could fill up an expensive SSD target. Or if the admin knows all log files are small, just go ahead and keep them exclusively on SSDs for small write performance!) And I guess "create_append_stripe_width=N" for completeness.
Update: Upon re-reading Andreas' comment, maybe the intent was to still use a PFL layout. That brings to mind a possibility of just defining a "create_append_extents=N" option which indicates how many of the default layout PFL extents to define. I think in our case we would set it to 2, to get first part on SSD and have a backstop of single larger stripe on HDD. The end of the last default extent (typically "-1") should be applied to the file, OR *_max_mb if that is defined but I think that would no longer be needed.
Update 2: Thanks Patrick for catching that I said "instantiate" when I should not have. My thought was to still create a special layout, with PFL, but base it on the default layout... truncated to N extents. Then the whole thing can be instantiated.
Thanks!
Mike,
could you please take a crack at this. It may be fairly easy for you to implement the basic functionality.
The general consensus is that we should default to create a 1-stripe file when a file is opened with O_APPEND, because there is purely overhead when having multiple stripes in this case due to extra OST object creation, extra locking of all the objects, etc. and no benefits (multiple threads cannot write to the same file anyway). There should be a tunable parameter that disables this feature, something like lctl set_param mdt.*.append_onestripe_enable=0 or ...append_logfile_enable=0 or similar to disable it (enabled by default since this is the most common case).
There could be a mdt.*.append_*_max_mb=N tunable to set the maximum size of the log files. Probably the default for *_max_mb should be unlimited (it will run out of space when OST is full), and use a plain file layout, which provides maximum compatibility for clients and applications. If it is not unlimited, then it will need to create a single PFL component with a limited size, and all PFL-capable clients will return an error if they try to write beyond the end of the component. There definitely may be a benefit in many environments to limit the maximum size of a log file to protect from a runaway job spamming the filesystem (I've wanted this even for local filesystems on occasion).
I was thinking that having a DoM + 1-stripe OST PFL layout could be useful, but this doesn't help in the end because O_APPEND will always instantiate the OST object anyway, and the OST would need to be checked for data each time. That would just be overhead and doesn't make any sense to do in the end compared to a DoM-only file.
However, I think it would make sense to allow creating a DoM-only PFL file if only small log files are used (which seems to be typical), as the lower write latency could improve application performance if they are blocked on the log writes. It could also hurt performance if the MDS is overloaded, and it could consume too much MDT space if the logs got very large, so this behavior shouldn't be enabled by default. Maybe mdt.*.append_*_enable=mdt to enable creating the component on DoM, and change *_max_mb by default (if currently unlimited) for such files to 32MB?
If we wanted to get fancy at some point in the future (if this feature gains interest), we could allow ...append_enable=[FID] together with patch https://review.whamcloud.com/28972 "LU-9982 lustre: clients striping from mapped FID in nodemap" to specify a source FID for the layout template for all O_APPEND files, which allows maximum flexibility instead of trying to specify a variety of layouts via tunable parameters, but this would require that LU-9982 is completed first to allow specifying an arbitrary FID as a layout template instead of the parent directory, so it isn't a goal for the initial patch.
This issue appears to be causing problems for us as well, primarily for Slurm stdout/stderr files (which, as Stephanie said, can be scattered all over various directories). Using 2.10.5+ servers and 2.10.7+ clients.
The other trigger that I don't see mentioned is probably less common... a seek to some point in the file without actually writing anything. It may not be directly related, but would be an opportunity for efficiency if any changes could also handle "dd if=/dev/zero of=sparse_file bs=1 count=0 seek=100G".
Sorry for the lack of context! Yes the above units are in bytes. I sometimes use histogram.py (from https://github.com/bitly/data_hacks ) to display quick histograms from data points, hence the weird ranges - they depend on the input data).
Anyway, yes, these files are all very small (< 128KB). Thanks for the different links, this is very interesting. To be honest, we would be happy already with a simple patch to try the one-stripe option, as long as there is a tunable to enable/disable on the flight. This would allow us to still use "large" PFL layout in general but not waste inodes on OSTs for O_APPEND files.
It isn't totally clear from the output, but are the above units in bytes (with strange fractional-byte boundaries) or KB?
In any case, if all or a large majority of O_APPEND files are smaller than 128KB, it seems to validate my theory that creating a one-stripe file when a file is first opened with O_APPEND would be perfectly fine. If these files are only logs/errors they will very rarely be larger than a few MB in size, and easily handled by one OST. That would be a very simple heuristic to implement (maybe with a tunable option to disable it if it not wanted), and would be easily backported if needed. The other proposals will need a considerably larger effort (changes to locking, CLIO layer, protocol compatibility) and would be much less likely to be available in an older release.
There is the patch https://review.whamcloud.com/33126 "LU-11234 lod: add uid/gid/suffix/jobid/nid pool selection layout_policy" which allows an administrator to specify an OST pool to be selected based on process/user/filename/client attributes. I think it would be much more useful to allow specifying a whole layout like patch https://review.whamcloud.com/28972 "LU-9982 lustre: Clients striping from mapped FID in nodemap", but at least it is a step in the right direction. This could also be improved to detect whether the
Andreas, please find below the size stats for 177,478 files found like that (partial scan done last night):
# NumSamples = 177478; Min = 0.00; Max = 129292.00 # Mean = 3339.485897; Variance = 60982229.732148; SD = 7809.111968; Median 1221.000000 # each ∎ represents a count of 2179 0.0000 - 12929.2000 [163461]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (92.10%) 12929.2000 - 25858.4000 [ 12062]: ∎∎∎∎∎ (6.80%) 25858.4000 - 38787.6000 [ 630]: (0.35%) 38787.6000 - 51716.8000 [ 396]: (0.22%) 51716.8000 - 64646.0000 [ 260]: (0.15%) 64646.0000 - 77575.2000 [ 235]: (0.13%) 77575.2000 - 90504.4000 [ 157]: (0.09%) 90504.4000 - 103433.6000 [ 116]: (0.07%) 103433.6000 - 116362.8000 [ 85]: (0.05%) 116362.8000 - 129292.0000 [ 76]: (0.04%)
All of them would fit in our "mdt" 128kB DOM stripe... but they have a PFL layout fully instantiated instead.
That would only happen if the first append was writing to a region where there was an uninstantiated component. As previously described, I think the components for the append should always be pre-instantiated before write starts.
At a minimum the full range of [size,size+len], which is known in advance, should be instantiated. Optionally, something like [size,size+N\*len], where N is something reasonable like 8-16, num_cpus, etc. so that racing appends are unlikely to need to retry. Even then, it may not be worthwhile to have N > 1 if one considers "len <<< component_size" so the chance of crossing a component is very small on any given write.
Andreas, no, I don't have a size distribution for O_APPEND files, sorry. The users scatter and name their slurm job output files in various ways so I would have to scan the whole file system and guess at naming conventions. And even that might not catch other files that happened to be created with O_APPEND.
If you can suggest a way to scan the filesystem for files which are not using their last instantiated extent, I'm happy to try to provide more data.
Capping the size on O_APPEND files is potentially useful, but also violates the principle of least surprise on a POSIX-like filesystem, and would lead to very unhappy and confused users if writes fail unexpectedly. Hence my suggestion of "truncating" the PFL layout to N extents, and keeping the extent end of the last component. Hopefully it would be fairly easy to take the layout that would otherwise be created on O_APPEND and just set the layout to the first N components, modifying the last component end to be the original layout's last component end. No additional pool specification needed, no max size limit surprises, the system "just works" for the users.