What is 2.3 SP1 and SP2? Do you see this with the 2.3 branch in the git tree? Or with Master?
Keith I think that Vitaly means SLES11 SP1 and SP2
this is not correct to kick transient pages off the CLIO cache, because we may get 2 sets of pages in osc_build_req(), cacheable and transient ones, for the same offsets.
http://review.whamcloud.com/6096 - revert of LU-481
http://review.whamcloud.com/6097 - fix of original LU-481 issue
lustre 2.3 on sles11 sp1/sp2
So can you please describe the scenario to hit this and the environment?
due to putting out transient pages of the page cache, we may get 2 sets of pages, cacheable and transient, for the same offsets, originated in BIO and DIO. later, when preparing rpc, such pages are gathered for the same rpc, and assertion that 2 pages with the same offset are found is hit.
Actually, I just started refresh the fix of LU-247 (http://review.whamcloud.com/#change,980) yesterday, which could fix this problem (concurrent dio read & buffered read), but that isn't a small fix, I'm not sure if it can be merged in 2.4.
Vitaly, can you please tell us what the workload is that is triggering this problem? Is this a synthetic workload that is generating "difficult" IO patterns with O_DIRECT and buffered IO on the same file offset, or is this a failure seen by a real application running from a user?
I would be surprised if any real application will have an IO workload to trigger this, but then again application developers are known to do all kinds of strange things...
If this is only from a synthetic workload, then I'm inclined to drop it as a blocker for the 2.4.0 release, since this is extremely unlikely to be hit under real-world usage.
Andreas, this is not some real application workload. It is from an internal stress test.
Let's mark the transient page as special in IO engine so that it can't be merged with caching pages.
Jay,
transient pages already marked as special and live in CLIO cache until we have IO finished, and merging with buffered pages exist in CLIO from beginning, so we may say - that is broken for now. Per Nikita`s comments - he tried to present a flat file as offset range with clio pages if it's covered by lock. That allowed to re-stripe it easy just a change mapping a some offset range to new osc object.
from other view - OST uses a single page cache to store DIO and buffered IO so we have no differences (buffered IO may read DIO data without access to disk). so we have free to use same view on client side, have a single CLIO page cache to store a data before send over network. That way is better in case we will return to idea to create a p2p communication and client may send own data to new client without asking any data from a server. That is very usefull in excacale as extend buffer cache dramatically.
Because of it, i think we need to return to put transient pages into CLIO cache, but remember that is special pages and need disowned after IO finished and don't forget we need to stay in waiting until that data committed on OST side and don't removed a request from sending queue, otherwise we will hit a LU-1573. But until request in sending queue, we a free to read/write in same pages, to reduce a network usage.
transient pages already marked as special and live in CLIO cache until we have IO finished, and merging with buffered pages exist in CLIO from beginning, so we may say - that is broken for now.
It was. But we found a problem in LU-481 where a transient page can be in cache when trying to add a caching page. We could fix it by awaiting transient pages to go away but I didn't see any benefits there. Any idea?
Per Nikita`s comments - he tried to present a flat file as offset range with clio pages if it's covered by lock. That allowed to re-stripe it easy just a change mapping a some offset range to new osc object.
This turns out too optimistic. Restriping is a case of layout change and right now we have a complete solution for layout change. The safest way to change a file's layout is to clean up file's page cache in prior. Don't bother talking about performance as changing layout is a rare operation 
from other view - OST uses a single page cache to store DIO and buffered IO so we have no differences (buffered IO may read DIO data without access to disk). so we have free to use same view on client side, have a single CLIO page cache to store a data before send over network.
hmm.. I tend to think DIO shouldn't pollute cache in anyways - this is why it is called Direct IO. Even on the OST side, it should go directly into disks before replying the clients. Also it;s really dangerous to cache write of DIO on the OSTs.
Alexey, we've seen quite a few problems caused by putting 'transient' pages in the radix tree:
- Race of buffered io vs. dio described in this ticket &
LU-481; - Unable to support unalinged DIO. Lustre now can only do aligned DIO (buffer address, offset & write count must be page size aligned), the reason we have such unnecessary
restriction is that we put user pages (transient pages) in the radix tree. I think unaligned DIO is very important for DIO users and it also very helpful for lockless io;
All in all, DIO pages are managed by applications, I don't see any reason to store them in the radix tree. (add them in the tree before io then remove on io completion looks only increase overhead).
I've refreshed the patch of LU-247 (http://review.whamcloud.com/#change,980), but I'm not sure if it can catch the 2.4 window. I think Jingshan's idea of not merging 'transient' page with cached page into same RPC could be a temporariy solution if the patch can't fit in with the 2.4 schedule.
> It was. But we found a problem in LU-481 where a transient page can be in cache when trying to add a caching page. We could fix it by awaiting transient pages to go away but I didn't see any benefits there. Any idea?
Vitaly have fix for LU-481, it's simple. But with that fix we have avoid doubling IO stream in case mixing IO and (as i say before) we may read from a that pages without sending network requests.
>This turns out too optimistic. Restriping is a case of layout change and right now we have a complete solution for layout change. The safest way to change a file's layout is to clean up file's page cache in prior. Don't bother talking about performance as changing layout is a rare operation
why? if you change a layout - you just change a translation about offset in file, to data object and offset to object. main question in that case - have a stable point when old layout and new layout have a same contents.
but that is outside of that ticket.
> hmm.. I tend to think DIO shouldn't pollute cache in anyways - this is why it is called Direct IO. Even on the OST side, it should go directly into disks before replying the clients. Also it;s really dangerous to cache write of DIO on the OSTs.
by POSIX/SUS definition, DIO need bypass only pagecache, but don't see anything about raid cache, disk drive cache or other cache types...
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html
Direct I/O
Historically, direct I/O refers to the system bypassing intermediate buffering, but may be extended to cover implementation-defined optimizations.
> Race of buffered io vs. dio described in this ticket & LU-481;
That ticket isn't race. that ticket is result of "fix" in LU-481. When you kill finding a buffered page for DIO access you start able to put two CLIO pages in same offset and OSC send two pages in single bulk with same offset.
> Unable to support unalinged DIO. Lustre now can only do aligned DIO (buffer address, offset & write count must be page size aligned), the reason we have such unnecessary
That is big question. You need to read full page anyway on OST side and have full page write.
but unaligned write may be not at DIO case, so you will fix a particle case and not a generic.
Why it's don't fix in different way? like send a first page of buffer as part of LOCK ENQ request?
that is need to change a wire protocol, but will solve DIO and buffered unaligned write access.
(CLIO release page should be revised to ability to kill pages only with locks cover it, otherwise we have much more problems with readahead, single page reads inside a large up2date region and other)
as about lockless IO.. lock less IO more easy implement as direct access to osc_brw api or other ways but have similar to obdecho client.
> All in all, DIO pages are managed by applications, I don't see any reason to store them in the radix tree. (add them in the tree before io then remove on io completion looks only increase overhead).
if you don't plan to have it's in radix tree - why it's should be covered by CLIO page? kill these pages from CLIO that is reduce a complexity of CLIO (we will have a single page type).
http://review.whamcloud.com/#/c/10930 landed, so should this bug be closed?
Yes it should - thanks!