[LU-247] Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Affects Version/s: Lustre 2.0.0
• Severity: 3
• Bugzilla ID: 18,801
• Assignee: Niu Yawei (Inactive)
• Reporter: Niu Yawei (Inactive)

Description

Port the fix of bug 18801 to 2.1

Issue Links

• is related to: LU-3192 LBUG:(osc_request.c:1308:osc_brw_prep_request()) ASSERTION( i == 0 || pg->off > pg_prev->off )
• is related to: LU-13805 i/o path: Unaligned direct i/o
• is related to: LU-12429 Single client buffered SSF write is slower than O_DIRECT
• is related to: LU-4664 sync write should consume grant on client
• is related to: LU-4198 Improve IO performance when using DIRECT IO using libaio

Comments

[Patrick Farrell]

See LU-13805

[Cory Spitz]

If we follow Niu's suggestion to land change #980 and then follow it up, can it land now, or must we wait until after we branch for b2_5?

[Andreas Dilger]

I don't like the idea of adding new users of the obd_brw() APIs. These are pretty much obsolete on both the client and server, and we should be removing them entirely instead of adding new users.

[Niu Yawei (Inactive)]

The patch in http://review.whamcloud.com/980 uses obd_brw() API in osc_io_dio(), since it requires less code changes and easyer for landing, however, the best way to implement osc_io_dio() should be: introduce a new API which can naturally accept (file offset, bytes, offset in first page, incontiguous pages) and build io requests based on these information directly (brw_page should not be used for direct io, 'pshift' hacking should be eliminated), and osc_io_dio() calls the new API to build io reqeusts and sends them asynchronously.

The proper implementaion requires more code changes, let's fix it in a follow-up patch.

[Niu Yawei (Inactive)]

Patch is at http://review.whamcloud.com/980

[Peter Jones]

I think that we will defer this task for now

[Oleg Drokin]

Frankly I am not even sure why do we need radix trees or anything like that for directio or otherwise uncached pages.
It's a purely one-shot job anyway.

Lai, your description of patchless io partial page write in clio where unlocked page is marked as clean sounds totally broken.
Good thing there is no easy way to trigger this mode or we'd have another blocker on our hands.

[Jinshan Xiong (Inactive)]

I realize we should give it a try because this may be not a problem in clio. Clio has a finer grained lockless IO control, it can do lockless IO per ost object, instead of per file as b18.

[Jinshan Xiong (Inactive)]

Hi Niu,

I composed a patch to bypass cl_page cache for TRANSIENT pages at: http://review.whamcloud.com/#change,495, just for a reference.

[Jinshan Xiong (Inactive)]

Hi Niu,

Thanks for the points. Yes, this is a good chance for us to fix the directIO problem in one shot.

I still tend to do this upon the infrastructure of clio. Also the reason to have a radix tree in clio is for porting purpose - we've done a lot of work to decouple linux vfs/vm in both MDT and client stack.

Here is my idea: let's fix CPT_TRANSIENT page implementation to have arbitrary buffers for this kind of pages. However, this will not fit into cl_page cache any more which means stale data may be read if applications are doing regular IO and directIO on the same node. This may violate posix semantics a little bit, but I tend to think this is fine since linux kernel is doing the same thing.

I'm going to compose a draft patch for this. It should be easy.