[LU-11264] llapi_* routines demonstrate poor performance

• Type: Improvement
• Resolution: Unresolved
• Priority: Major
• Environment: Argonne's Theta machine is a Cray running lustre: 2.7.3.74 ; kernel: patchless_client ; build: 2.7.3.74

Epic/Theme

Performance

• Assignee: WC Triage
• Reporter: Robert Latham

Description

I'm going to bug Cray about this but Andreas said  "we can't fix problems that aren't in Jira" so here I am.

In the old days one would get striping information via an ioctl:

   

struct lov_user_md *lum;
    size_t lumsize = sizeof(struct lov_user_md) +LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data);
    lum->lmm_magic = LOV_USER_MAGIC;
    ret = ioctl( fd, LL_IOC_LOV_GETSTRIPE, (void *)lum );

 

 

Then the llapi_ routines showed up:

 

   ret = llapi_file_get_stripe(opt_file, lum);

and even more sophisticated llapi_layout_ routines:

 

struct llapi_layout *layout;
 layout = llapi_layout_get_by_fd(fd, 0);
 llapi_layout_stripe_count_get(layout, &(lum->lmm_stripe_count));
 llapi_layout_stripe_size_get(layout, &(lum->lmm_stripe_size));
 llapi_layout_ost_index_get(layout, 0, &(lum->lmm_stripe_offset));
 

 

I haven't done a full scaling run, but small scale tests don't look great for the newer interfaces.   Average time to get the stripe size for a 2 node 128 process MPI job:

ioctl:   17.2 ms

llapi_file_get_stripe: 142.0 ms

llapi_layout:  2220 ms

 

Wow!  I want to use the new interfaces but these overheads are bonkers.

 

Issue Links

• is related to: LU-14316 lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE), use llapi_layout_get_by_path()
• is related to: LU-14897 "lfs setstripe/migrate" should check if pool exists for PFL components
• is related to: LU-17025 Invalid pool name does not return error
• is related to: LU-10500 Buffer overflow of 'buffer' due to non null terminated string 'buffer' in llapi_search_tgt
• is related to: LU-12327 DNE3: add llapi_dir_create_from_xattr() and llapi_dir_default_from_xattr()
• is related to: LU-12369 lfs setstripe causes MDS DLM lock callback during create
• is related to: LU-14396 cache llapi_search_mounts() result
• is related to: LU-14645 setstripe cleanup
• is related to: LU-13693 lfs getstripe should avoid opening regular files