Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11264

llapi_* routines demonstrate poor performance

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      Argonne's Theta machine is a Cray running lustre: 2.7.3.74
      kernel: patchless_client
      build: 2.7.3.74
    • Epic/Theme:
    • Rank (Obsolete):
      9223372036854775807

      Description

      I'm going to bug Cray about this but Andreas said  "we can't fix problems that aren't in Jira" so here I am.

      In the old days one would get striping information via an ioctl:

         

      struct lov_user_md *lum;
          size_t lumsize = sizeof(struct lov_user_md) +LOV_MAX_STRIPE_COUNT * sizeof(struct lov_user_ost_data);
          lum->lmm_magic = LOV_USER_MAGIC;
          ret = ioctl( fd, LL_IOC_LOV_GETSTRIPE, (void *)lum );
      

       

       

      Then the llapi_ routines showed up:

       

         ret = llapi_file_get_stripe(opt_file, lum);

      and even more sophisticated llapi_layout_ routines:

       

      struct llapi_layout *layout;
       layout = llapi_layout_get_by_fd(fd, 0);
       llapi_layout_stripe_count_get(layout, &(lum->lmm_stripe_count));
       llapi_layout_stripe_size_get(layout, &(lum->lmm_stripe_size));
       llapi_layout_ost_index_get(layout, 0, &(lum->lmm_stripe_offset));
       
      

       

      I haven't done a full scaling run, but small scale tests don't look great for the newer interfaces.   Average time to get the stripe size for a 2 node 128 process MPI job:

      ioctl:   17.2 ms

      llapi_file_get_stripe: 142.0 ms

      llapi_layout:  2220 ms

       

      Wow!  I want to use the new interfaces but these overheads are bonkers.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wc-triage WC Triage
                Reporter:
                robl Robert Latham
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: