https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3062 Lustre <p>After we upgraded our clients from 2.1.3 to 2.3.0, some users (the crowd is increasing) started seeing their application to fail, to hang, or even crash. The servers run 2.1.4. In all cases, same application ran OK with 2.1.3.</p> <p>Since we do not have reproducer on the hang and the crash cases, we here attach a reproducer that can cause application to fail. The test were executed with stripe count of 1, 2, 4, 8, 16. The higher number the stripe count the more likely application fails.</p> <p>The 'reproducer1.scr' is a PBS script to start 1024 mpi tests.<br/> 'reproducer1.scr.o1000145' is PBS output of the execution.<br/> '1000145.pbspl1.0.log.txt' is an output of one of our tools to collect /var/log/messages from servers and clients related to the specified job.</p> <p>The PBS specific argument lines start with "#PBS " string and are ignored if executed without PBS. The script use SGI MPT, but can be converted to openmpi or intel mpi.</p> Lustre server 2.1.4 centos 6.3<br/> Lustre clients 2.3.0 sles11sp1 LU-3062

Multiple clients writing to the same file caused mpi application to fail

Bug Major Resolved Cannot Reproduce Oleg Drokin Jay Lan ptr Fri, 29 Mar 2013 00:54:55 +0000 Thu, 8 Sep 2016 21:33:28 +0000 Thu, 8 Sep 2016 21:33:28 +0000 Lustre 2.3.0 0 9 <p>Ok, so from the logs we can see the client was evicted by server for some reason.<br/> Now why it was evicted is not clear because there seems to be no server logs included, but I imagine it's due to AST timeouts. We included multiple patches in 2.4 to help this cause.</p> <p>In addition to that I cannot stop wondering about this message:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Thu Mar 28 12:43:49 2013 M r325i4n13 kernel: [569506.266405] pcieport 0000:00:02.0: AER: Multiple Corrected error received: id=0010 Thu Mar 28 12:43:49 2013 M r325i4n13 kernel: [569506.266572] pcieport 0000:00:02.0: PCIE Bus Error: severity=Corrected, type=Data Link Layer, id=0010(Receiver ID) Thu Mar 28 12:43:49 2013 M r325i4n13 kernel: [569506.276986] pcieport 0000:00:02.0: device [8086:3c04] error status/mask=00000040/00002000 Thu Mar 28 12:43:49 2013 M r325i4n13 kernel: [569506.285434] pcieport 0000:00:02.0: [ 6] Bad TLP </pre> </div></div> <p>hopefully it did not result in any dropped messages.</p> <p>About the "PCIE Bus Error: severity=Corrected" errors, I checked with our admins. He said it was normal and not indicative of an IB problem.</p> <p>I will collect and provide logs at the servers.</p> <p>BTW, the test was run using 1024 cpus distributed to 64 nodes. However, I was able to reproduce the problem with only 4 sandy bridge nodes, 4*16=64 processes.</p> <p>The PCIE corrected errors seem to be related to sandy bridge PCI 3.0. We have seen 10,000s of those errors a day. However, same applications did not fail if run 2.1.3 clients.</p> <p>Can you identify those AST timeouts patches? Are they client side? Note that we run 2.1.4 at servers. Is there any issue with 2.1.4 server + 2.3.0 client combination?</p> <p>This tarball contains syslog between <span class="error">&#91;Thu Mar 28 12:00:00 2013&#93;</span> and <span class="error">&#91;Thu Mar 28 13:00:00 2013&#93;</span>.</p> <p>service160 is mds/mgs. The rest are oss'es.</p> <p>Hi Jay Lan,</p> <p>What does "write(9999) 66" mean in the reproducer? I mean how much data it will write to the file by this command.</p> <p>Can you please collect lustre logs on the client and server side while running the reproducer?</p> <p>(pbspl1,241) od -x fort.9999<br/> 0000000 0004 0000 0042 0000 0004 0000<br/> 0000014<br/> (pbspl1,242) ls -l fort.9999<br/> <del>rw-r</del><del>r</del>- 1 jlan g1099 12 Mar 28 18:39 fort.9999<br/> (pbspl1,243) </p> <p>It is 12 bytes.</p> <p>Hi Jinshan,</p> <p>The client side logs are in 1000145.pbspl1.0.log.txt. You may want to filter out pbs information. The nbp2-server-logs.<a href="https://jira.whamcloud.com/browse/LU-3062" title="Multiple clients writing to the same file caused mpi application to fail" class="issue-link" data-issue-key="LU-3062"><del>LU-3062</del></a> is the tarball of all server side logs:</p> <p>linux39.jlan 109&gt; tar -tzf nbp2-server-logs.<a href="https://jira.whamcloud.com/browse/LU-3062" title="Multiple clients writing to the same file caused mpi application to fail" class="issue-link" data-issue-key="LU-3062"><del>LU-3062</del></a><br/> service160<br/> service161<br/> service162<br/> service163<br/> service164<br/> service165<br/> service166<br/> service167<br/> service168</p> <p>Service160 is the mds/mgs, the rest are oss.</p> <p>This is a shorten version of 1000145.pbspl1.0.log.txt with pbs_mom and epilogue messages removed.</p> <p>Hi Jay Lan,</p> <p>I already took a look at those files, and I need more detail information about, can you please turn on more debug options, especially LNET on the client and server side and collect it again? The most interesting thing is that even the clients lost connection to the MGS which is not involved in the IO path at all. If I guess it correctly, this is likely a LNET problem. But I'd like to make it clear before pointing my finger to others.</p> <p>Do you know if f90 opens the file with O_APPEND, and "write(9999) 66" just writes 66 bytes to the file?</p> <p>THank you.</p> <p>I will try to reproduce the problem with increased debugging.</p> <p>The f90 program does not open with O_APPEND. All instances writes 12 bytes to the file. The content of the file:<br/> 0000000 0004 0000 0042 0000 0004 0000<br/> contains three 4-byte word. The first and the last words probably are envelop. The second byte is the hex value of the number "66". Not 66 bytes. The program just write a number "66" to the output file.</p> <p>original reproducer had a bug in it.</p> <p>Client debug logs. Server will be tougher to get. We may have to switch to our test filesystem to get that to work. Please look at client side logs and determine if you want me to still get server logs.</p> <p>Is there an interop issue between 2.3.0 client and 2.1.4 server? Any change in 2.3.0 client requires same change at the server?</p> <p>full debug turned on this time.. debug logs were over 500MB.. jira has only 10MB limit so i took the portion of the log that made sense, that shows the OST disconnect. let me know if you want the whole client logs and we can figure out how to get them to you.</p> <p>Uploading full debug log file. Split file first, then compressed each segment using bzip2 to get below the max file size of 10MB.<br/> Order of files is xaa, xab, xac, xad, xae.</p> <p>Bobbie</p> <p>Could you please setup the reproducer supplied on April 1st above.</p> <p>Thanks</p> <p>Peter</p> <p>I have received my account on Rosso and expect to complete the testing over the coming week.</p> <p>So, just to clarify, the problem here is that the reproducer program is starting 1024 tasks to write 12 bytes to the same offset=0 of the same file (striped over 16 OSTs?), and there is a lot of contention? Or am I misunderstanding and each thread will write to non-overlapping ranges of the file (i.e. like O_APPEND)?</p> <p>This isn't terribly surprising, because either case is a pathologically bad IO pattern. If they are all writing to the same offset it is completely serialized by the locking, while using O_APPEND actually gets worse with increasing numbers of stripes, since it needs to lock all stripes to get the current file size.</p> <p>Do you have any idea what the application is actually trying to accomplish with these overlapping writes? Is there any chance to modify the application to do (whatever it is trying to do) in a more sensible manner? Depending on what the application is actually trying to accomplish, there may be many more filesystem-friendly ways of doing this.</p> <p>The servers should definitely not fail in this case, though I can imagine that the clients might time out waiting for their chance to overwrite the same bytes again. The clients should reconnect and complete the writes, however.</p> <p>It might be possible to optimize pathological cases like this by using OST-side locking for the RPCs, though there is still a difficulty with sending sub-page writes that also need to be handled.</p> <p>Sent Andreas email on 18th, but didn't add to Jira.</p> <p>Hey Andreas,</p> <p>Our position is that users doing this type of work should<br/> not cause an eviction. We agree that it is sub-optimal<br/> at best (we have a different term for it: stupid), but<br/> our users continue to do it. There are various reasons<br/> why users can't/won't change their code here at NASA.<br/> Word from management is that we need to get this fixed.<br/> I've copied our local Lustre team in case anyone has<br/> anything else to add.</p> <p>Thanks,</p> <p>jdk</p> <p>I'm attaching the logs from running the reproducer on the opensfs cluster. It was a 16 node, 1mds, 2oss test.</p> <p>It seems that this is due to a deadlock on the client side.</p> <p>Is it possible to do an experiment as follows:<br/> 1. write 66 to the shared file as usual;<br/> 2. if the write can't be finished in 5 minutes(this is only true on NASA cluster because the lock timeout was 1900 seconds; on our local site, we should make it shorter, say 60 seconds), the script will dump the stack trace of processes running on the node by 'echo t &gt; /proc/sysrq-trigger'.</p> <p>After taking a further look, this should be a problem about truncate because the test program was opening the file with O_TRUNC. This is why more stripes leads to more likely causing the problem.</p> <p>Can you please try this patch: <a href="http://review.whamcloud.com/5208" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5208</a> to see if it can help?</p> <p>Jim Karellas tested 2.4 client (against 2.1.5 server) before and he was<br/> still able to reproduce the problem with ease. Since 2.4 contains the <a href="http://review.whamcloud.com/5208" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5208</a> patch Jinshan mentioned and wanted us to test, I think we need to feed back the test result.</p> <p>We have not seen this for a long while and we are running 2.7, so please close it.</p> <p>ok Jay</p> Development End date Thu, 18 Jul 2013 00:54:55 +0000 Rank 1|hzvmmv: Rank (Obsolete) 7461 Severity Start date Fri, 29 Mar 2013 00:54:55 +0000