[LU-6737] many stripe testing of DNE2

• Type: Task
• Resolution: Fixed
• Priority: Blocker
• Assignee: Robert Read
• Reporter: Richard Henwood (Inactive)

Description

Many stripe count test

The many stripe count functional test is intended to show that a DNE2 configuration can handle many MDTs in a single filesystem, and a single directory can be striped over many MDTs. Due to the virtual AWS environment in which this is being tested, while performance will be measured, neither performance scaling nor load testing are primary goals of this test. It is rather a functional scaling test of the ability of the filesystem configuration and directory striping code to handle a large number of MDTs.

1. Create a filesystem with 128 MDTs, 128 OSTs and at least 128 client mount points (multiple mounts per client)
2. Create striped directories with stripe count N in 16, 32, 64, 96, 128:

           lfs setdirstripe -c N /mnt/lustre/testN
   

   Note: This command creates a striped directory across N MDTs.

           lfs setdirstripe -D -c N /mnt/lustre/testN
   

   Note: This command sets the default stripe count to N. All directories created within this directory will have this default stripe count applied.

3. Run mdtest on all client mount points, and each thread will create/stat/unlink at least 128k files in the striped test directory. Run this test under a striped directory with default stripes, so all of subdirectories will be striped directory.

           lfs setdirstripe -c N /mnt/lustre/testN
           lfs setdirstripe -D -c N /mnt/lustre/testN
   

4. No errors will be observed, and balanced striping of files across MDTs will be observed.

Issue Links

• is blocked by: LU-6602 ASSERTION( rec->lrh_len <= 8192 ) failed
• is blocking: LU-6858 Demonstrate DNE2 functionality

Comments

[Robert Read]

As I already have some automation built around mdsrate, I'll use that and ensure all threads use a single, shared directory. I also have a patch to mdsrate that adds support for directory striping, though I'll use the lfs commands to make it explicit.

I'll add an `lfs df -i` before and after the create step of each test so we can confirm, at least manually, that the files are balanced.

Do you want the files created with 0-stripes or normally with a single stripe?

BTW, I've been putting our provisioning tools through the paces today, but thought I'd try one client with 128k files. I noticed that large DNE striping has a pretty big impact on directory scanning performance:

[ec2-user@client00 ~]$ lfs getdirstripe /mnt/scratch/test/dir-0/
/mnt/scratch/test/dir-0/
lmv_stripe_count: 0 lmv_stripe_offset: 0

[ec2-user@client00 ~]$ time lfs find /mnt/scratch/test/dir-0/ |wc -l 
131073

real	0m0.112s
user	0m0.038s
sys	0m0.159s

[ec2-user@client00 ~]$ time lfs getdirstripe /mnt/scratch/test128/dir-0/ 
/mnt/scratch/test128/dir-0/
lmv_stripe_count: 128 lmv_stripe_offset: 0
[stripe details deleted]

[ec2-user@client00 ~]$ time lfs find /mnt/scratch/test128/dir-0 |wc -l
131073

real	0m43.969s
user	0m0.053s
sys	0m43.990s

I noticed this because "lfs getdirstripe" was taking ~40s to return because for some reason lfs reads all the directory entries after printing the striping data. I'll make sure to only do this on empty directories for now.

[Di Wang (Inactive)]

No, it is not required. i.e. lfs setdirstripe will set default stripes on both normal dir and striped directory.

[Robert Read]

Is it required for testN to be a striped directory in order set the default to be striped? In other words, would the following result in striped subdirectories of testN:

    mkdir /mnt/lustre/testN
    lfs setdirstripe -D -c N /mnt/lustre/testN

[Di Wang (Inactive)]

 
lfs setdirstripe -c N /mnt/lustre/testN
lfs mdkir -c N /mnt/lustre/testN

They are same, and they will create a striped directory with stripe_count =2. Note: if you do not indicate -i here, the master stripe(stripe 0) will be in the same MDT with its parent.

lfs setdirstripe -D -c N /mnt/lustre/testN

This will be used to set default stripe count of testN, i.e. all of the subdirectories under testN will be created with this layout (-c N). And also the default stripe will be inherited by these subdirectories as well.

[Robert Read]

What is the difference between these three commands?

        lfs setdirstripe -c N /mnt/lustre/testN
        lfs setdirstripe -D -c N /mnt/lustre/testN
        lfs mdkir -c N /mnt/lustre/testN

[Andreas Dilger]

Since this is mostly intended to be a functional test of how many MDTs the DNE2 code can use instead of a performance test, there is a large leeway in terms of the testing options. I don't have a strong preference for mdtest over mdsrate, with the minor caveat that mdsrate is a Lustre-specific benchmark while mdtest is not. The goal would be to create all of the files in the one striped directory, rather than having each client/thread create its own subdirectory.

There could be multiple threads per client mountpoint, since even without the multi-slot last_rcvd patches or other workarounds there can be one RPC in flight per MDT so this would also provide natural scaling at the client as the number of MDTs increases.

As for determining the MDT load balance, given the large numbers of files and the fact that these are newly formatted filesystems I think that lfs df -i before and after each test would be enough to determine whether the created files are roughly evenly distributed across MDTs or not. Since the MDT selection is done via a hash function, the distribution should be fairly even but not perfectly so. Ideally, if you already have infrastructure in CE to monitor MDS load (e.g. LMT) then it would be interesting to see if the load is distributed evenly across MDSes during runtime, but that is not a requirement for this testing since it is targeted at testing the limits of MDSes and MDTs counts.

"No errors" means at a minimum no application-visible errors. For a purely functional test like this I would also expect that there are no Lustre-level errors either (timeouts, etc). If any Lustre errors are printed during the test run please attach them here, or create a new ticket if they indicate some more serious problem.

As for 128000 vs 131072 I don't think it really matters - the goal is to create a decent number of files per MDT to ensure a reasonable minimum runtime without creating so many that the tests with low MDT counts take too long. Creating 128 * 128000 files = 16M files, which would likely be too many for a 1-stripe directory, but should be reasonable for 16+ stripes (~1M/MDT at 16 stripes, down to ~128K/MDT at 128 stripes) which is the minimum for this test unless the hash distribution is totally broken.

[Robert Read]

Can mdsrate be used instead of mdtest? If mdtest is required, please post a script to run this as I don't have one for mdtest.

Should the stats and unlinks be done on different client nodes than the one that created them?

Just one thread per client mount point?

Please provide a method to determine if directory striping is balanced. Since test requires unlink, this need to be integrated into the test loop and ideally could be done using MPI, too.

Does "No errors" just mean no application errors? Or does this also mean there should be no lustre messages printed on any any of the consoles during the test run?

Just to be precise, does 128k mean 12800 or 2^17 (131072)?