Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Lustre 2.3.0
-
Lustre server 2.1.4 centos 6.3
Lustre clients 2.3.0 sles11sp1
-
2
-
7461
Description
After we upgraded our clients from 2.1.3 to 2.3.0, some users (the crowd is increasing) started seeing their application to fail, to hang, or even crash. The servers run 2.1.4. In all cases, same application ran OK with 2.1.3.
Since we do not have reproducer on the hang and the crash cases, we here attach a reproducer that can cause application to fail. The test were executed with stripe count of 1, 2, 4, 8, 16. The higher number the stripe count the more likely application fails.
The 'reproducer1.scr' is a PBS script to start 1024 mpi tests.
'reproducer1.scr.o1000145' is PBS output of the execution.
'1000145.pbspl1.0.log.txt' is an output of one of our tools to collect /var/log/messages from servers and clients related to the specified job.
The PBS specific argument lines start with "#PBS " string and are ignored if executed without PBS. The script use SGI MPT, but can be converted to openmpi or intel mpi.
Attachments
Activity
So, just to clarify, the problem here is that the reproducer program is starting 1024 tasks to write 12 bytes to the same offset=0 of the same file (striped over 16 OSTs?), and there is a lot of contention? Or am I misunderstanding and each thread will write to non-overlapping ranges of the file (i.e. like O_APPEND)?
This isn't terribly surprising, because either case is a pathologically bad IO pattern. If they are all writing to the same offset it is completely serialized by the locking, while using O_APPEND actually gets worse with increasing numbers of stripes, since it needs to lock all stripes to get the current file size.
Do you have any idea what the application is actually trying to accomplish with these overlapping writes? Is there any chance to modify the application to do (whatever it is trying to do) in a more sensible manner? Depending on what the application is actually trying to accomplish, there may be many more filesystem-friendly ways of doing this.
The servers should definitely not fail in this case, though I can imagine that the clients might time out waiting for their chance to overwrite the same bytes again. The clients should reconnect and complete the writes, however.
It might be possible to optimize pathological cases like this by using OST-side locking for the RPCs, though there is still a difficulty with sending sub-page writes that also need to be handled.
I have received my account on Rosso and expect to complete the testing over the coming week.
Bobbie
Could you please setup the reproducer supplied on April 1st above.
Thanks
Peter
Uploading full debug log file. Split file first, then compressed each segment using bzip2 to get below the max file size of 10MB.
Order of files is xaa, xab, xac, xad, xae.
full debug turned on this time.. debug logs were over 500MB.. jira has only 10MB limit so i took the portion of the log that made sense, that shows the OST disconnect. let me know if you want the whole client logs and we can figure out how to get them to you.
Is there an interop issue between 2.3.0 client and 2.1.4 server? Any change in 2.3.0 client requires same change at the server?
Client debug logs. Server will be tougher to get. We may have to switch to our test filesystem to get that to work. Please look at client side logs and determine if you want me to still get server logs.
I will try to reproduce the problem with increased debugging.
The f90 program does not open with O_APPEND. All instances writes 12 bytes to the file. The content of the file:
0000000 0004 0000 0042 0000 0004 0000
contains three 4-byte word. The first and the last words probably are envelop. The second byte is the hex value of the number "66". Not 66 bytes. The program just write a number "66" to the output file.
Hi Jay Lan,
I already took a look at those files, and I need more detail information about, can you please turn on more debug options, especially LNET on the client and server side and collect it again? The most interesting thing is that even the clients lost connection to the MGS which is not involved in the IO path at all. If I guess it correctly, this is likely a LNET problem. But I'd like to make it clear before pointing my finger to others.
Do you know if f90 opens the file with O_APPEND, and "write(9999) 66" just writes 66 bytes to the file?
THank you.
Sent Andreas email on 18th, but didn't add to Jira.
Hey Andreas,
Our position is that users doing this type of work should
not cause an eviction. We agree that it is sub-optimal
at best (we have a different term for it: stupid), but
our users continue to do it. There are various reasons
why users can't/won't change their code here at NASA.
Word from management is that we need to get this fixed.
I've copied our local Lustre team in case anyone has
anything else to add.
Thanks,
jdk