|
Robert,
I'm a Cray person, but I'll discuss this here so as not to leave the Whamcloud guys in the cold.
What exactly are you measuring? Is this execution time of the function directly, within a binary, or is it execution time of an entire binary containing just this function call?
Can you share whatever your test code is here? Just to make our lives easier.
A few more thoughts/questions.
- The llapi_file_get_stripe function you selected is not a direct equivalent of that ioctl. It also has to open() the file. So that may be a source of some of the overhead.
- Which of the llapi_layout functions is taking that much time? You don't specify.
Those last three functions (stripe_count_get, stripe_size_get, and ost_index_get) literally just access pointers in the memory given, so if they're taking milliseconds by themselves...
|
|
Thanks, Rob - I actually just grabbed your test and did a bunch of testing myself.
I see a really wildly large gap even with 1 process - interesting.
This is with debug turned on, on a small VM setup:
[root@cent7c01 cent7s01]# rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -i testfile; lctl dk > /tmp/out3
opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 0, opt_ioctl: 1, opt_fpp: 0, opt_llapi_fd: 0, nprocs: 1, time: 2.622000 ms
[root@cent7c01 cent7s01]# rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -a testfile; lctl dk > /tmp/out
opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 1, opt_ioctl: 0, opt_fpp: 0, opt_llapi_fd: 0, nprocs: 1, time: 3.715000 ms
[root@cent7c01 cent7s01]# rm -f testfile; lctl clear; mpirun --allow-run-as-root -n 1 /tmp/a.out -c -A testfile; lctl dk > /tmp/out2
opt_file: testfile, opt_create: 1, opt_fstat: 0, opt_lseek: 0, opt_realpath: 0, opt_llapi: 0, opt_ioctl: 0, opt_fpp: 0, opt_llapi_fd: 1, nprocs: 1, time: 5.026000 ms
So that's "ioctl" at 2.6 ms
Get_stripe at 3.7 ms
And
get_by_fd at 5.0 ms
That's... wild. I'll try to dig in later.
EDIT:
My previous comment about no additional server communications was wrong; had an error in my data collection.
The simple create+ioctl does 2, one of which is close().
The llapi_get_stripe function does 3 (create, some sort of get, then close).
The layout get function does 4 .
I've got some ideas why - I'll take a further look next week.
|
|
Robert,
Here's the problem in a nutshell.
lfs getstripe asks the client for the stripe count information that it has locally, and gets it locally if it's present. This is fast because that information is generally already cached locally when the file is open.
The other two functions do not ask the client for the layout, as such - Instead, the ioctl they use actually directly reads the layout extended attribute from the MDS. This read of an extended attribute cannot be satisfied by the layout information held in memory on the client (which it usually has if the file is open), instead, it goes directly to the MDS.
The solution would be to rewrite the implementation of those new ioctls to get the layout that the client has in memory and request it if it's not present.
|
|
I just happened to be looking at this code on the setstripe side and see a number of places where it could be optimized. Running a simple test comparing lfs setstripe with touch and createmany shows a big difference in performance (files removed between each test):
It is pretty clear from the elapsed time and the system time that lfs setstripe is doing a lot more stuff for each file create compared to the others (touch is additionally setting the timestamps). Looking at the strace of lfs setstripe -c1 on a client shows what is going on:
Firstly, it appears that liblustreapi_init() function is always doing some work to initialize the PRNG, but this PRNG is only used in a small number of cases. This initialization can be skipped unless the PRNG is actually needed in particular functions, even though it is only a one-time thing it is still needless overhead.
More importantly, it is doing multiple traversals of the full pathname (bad) to do realpath() on the filename and processing /proc/mounts for each file to determine if the file is on a Lustre filesystem or not. IMHO, it should be enough to return an error from ioctl(LL_IOC_LOV_SETSTRIPE) to determine that the file is not on a Lustre filesystem, or if that is really deemed necessary we can do statfs() on the directory and check the magic without doing dozens of syscalls.
There is also extra overhead if a pool name is specified to verify that it is valid that could be a one-time check for repeated use of the same pool(s), and to verify the stripe_size >= PAGE_SIZE, where again the PAGE_SIZE does not change from one file to the next.
Finally, I observe in the debug logs that between open(O_LOV_DELAY_CREATE) and ioctl(LL_IOC_LOV_SETSTRIPE) there is a DLM lock callback from the MDS to the client for the layout lock (I think). It doesn't make sense to grant the MDS_INODELOCK_LAYOUT lock to the client opening with O_LOV_DELAY_CREATE, when we know the client will immediately be changing it.
|
llapi-perf.c
lustre-ioctl-vs-llapi.png