[LU-4931] New feature of giving server/storage side advice of accessing file

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Labels: p4j patch
• Assignee: Li Xi (Inactive)
• Reporter: Li Xi (Inactive)

Description

We implement a new feature which provides new APIs and utils for senior users and smart applications to give advices about the access pattern of Lustre file so as to improve the data/metadata access peformance. It has a similar idea with fadvise64_64(2) or posix_fadvise(2), yet can pass specical advices directly through Lustre client to server/storage side.

Some tests show that this feature might help us to get performance improvement for some application by giving proper advices in advance.

Issue Links

• has to be finished together with: LUDOC-327 documentation for ladvise feature
• is related to: LU-6254 Fix OFD/OSD prefetch for osd-ldiskfs and osd-zfs
• is related to: LU-5561 Lustre random reads: 80% performance loss from 1.8 to 2.6
• is related to: LU-8565 sanity test 255a fails with ‘Speedup with willread is less than X%, got Y%’
• is related to: LU-6671 Wireshark: LDLM_ENQUEUE reply with undecoded buffer
• is related to: LU-8591 allow specifying ZFS blocksize via ladvise
• is related to: LU-8902 Add testing for the llapi_ladvise() API
• is related to: LU-7225 change ladvise wire protocol for lockahead and future usage
• is related to: LU-137 ioctl passthrough mechanism for Lustre OST/MDT mountpoints

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/21940/
Subject: LU-4931 ladvise: add code for ladvise_hdr into wirecheck.c
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5ee1287305fb6b6c472d097ef9a86a9e315104e4

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/22489/
Subject: LU-4931 ofd: use thread buffer for ladvise
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c29cf72acd431e65f0438804561e7c30feef0366

[Peter Jones]

Man page has landed. Remaining patches tracked under this id will be landed under a new ticket

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/22910/
Subject: LU-4931 doc: update ladvise man page
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f63e53a364f5162c0f8a81e42978c5a2b9b7522d

[Andreas Dilger]

Reopen to land man page for 2.9.0:

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22910
Subject: LU-4931 doc: update ladvise man page
Project: fs/lustre-release
Branch: master
Current Patch Set: 2
Commit: 61ceb43c858d6fc979fc4da9d2a925026b27859a

[Peter Jones]

As per discussion with Ihara all further enhancements in this area will be tracked under a different JIRA ticket

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20203/
Subject: LU-4931 ladvise: Add dontneed advice support for ladvise
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a5a7890093ea2509db15f8aa8a8c9d9c86133209

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22489
Subject: LU-4931 ofd: use thread buffer for ladvise
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 64b359ee1aa58b95c13323039b300a04556ff033

[Andreas Dilger]

Also, while you are in there, can you please fix the indenting for

static int ofd_ladvise_hdl(struct tgt_session_info *tsi)
{
        :
        :
                case LU_LADVISE_WILLREAD:
                        req->rq_status = ofd_ladvise_prefetch(env, 
                                fo,
                                ladvise->lla_start,
                                ladvise->lla_end);

to be

                        req->rq_status = ofd_ladvise_prefetch(env, fo,
                                                        ladvise->lla_start,
                                                        ladvise->lla_end);

[Andreas Dilger]

Li Xi, I noticed just now in ofd_ladvise_prefetch() that this is allocating PTLRPC_MAX_BRW_PAGES * sizeof(niobuf_local) = 160KB for each ladvise willread call. Instead, this should be using struct tgt_thread_big_cache *tbc = req->rq_svc_thread->t_data as is done in tgt_brw_read().