https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2218 Lustre <p>Sanger have been running into an issue where one of their applications seems to deadlock OSTs. They have an application that does lots of small IO and seems to create and delete a lot of files. It also seems to saturate the network, so there are a lot of bulk IO errors. It looks like the quota and jbd sections are getting into some kind of deadlock. I'm uploading the full logs, but there is a lot of:</p> <p>Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264411] <span class="error">&#91;&lt;ffffffff8139ba25&gt;&#93;</span> rwsem_down_failed_common+0x95/0x1e0<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264418] <span class="error">&#91;&lt;ffffffff8139bb8f&gt;&#93;</span> rwsem_down_write_failed+0x1f/0x30<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264425] <span class="error">&#91;&lt;ffffffff811e8db3&gt;&#93;</span> call_rwsem_down_write_failed+0x13/0x20<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264431] <span class="error">&#91;&lt;ffffffff8139ad8c&gt;&#93;</span> down_write+0x1c/0x20<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264438] <span class="error">&#91;&lt;ffffffff8114fd3f&gt;&#93;</span> dquot_initialize+0x8f/0x1c0<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264453] <span class="error">&#91;&lt;ffffffffa098fff0&gt;&#93;</span> ldiskfs_unlink+0x130/0x270 <span class="error">&#91;ldiskfs&#93;</span><br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264484] <span class="error">&#91;&lt;ffffffffa0a18a58&gt;&#93;</span> filter_vfs_unlink+0x2f8/0x500 <span class="error">&#91;obdfilter&#93;</span><br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264499] <span class="error">&#91;&lt;ffffffffa0a2c412&gt;&#93;</span> filter_destroy+0x1572/0x1b90 <span class="error">&#91;obdfilter&#93;</span><br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264512] <span class="error">&#91;&lt;ffffffffa09e4436&gt;&#93;</span> ost_handle+0x2f36/0x5ef0 <span class="error">&#91;ost&#93;</span><br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264539] <span class="error">&#91;&lt;ffffffffa06fb040&gt;&#93;</span> ptlrpc_main+0x1bc0/0x22f0 <span class="error">&#91;ptlrpc&#93;</span><br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264574] <span class="error">&#91;&lt;ffffffff81003eba&gt;&#93;</span> child_rip+0xa/0x20<br/> Oct 21 11:29:40 lus08-oss2 kernel: [ 1456.264577]</p> <p>and</p> <p>Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266346] Call Trace:<br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266366] <span class="error">&#91;&lt;ffffffffa0956006&gt;&#93;</span> start_this_handle+0x356/0x450 <span class="error">&#91;jbd2&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266388] <span class="error">&#91;&lt;ffffffffa09562e0&gt;&#93;</span> jbd2_journal_start+0xa0/0xe0 <span class="error">&#91;jbd2&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266398] <span class="error">&#91;&lt;ffffffffa095632e&gt;&#93;</span> jbd2_journal_force_commit+0xe/0x30 <span class="error">&#91;jbd2&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266415] <span class="error">&#91;&lt;ffffffffa0995ce1&gt;&#93;</span> ldiskfs_force_commit+0xb1/0xe0 <span class="error">&#91;ldiskfs&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266444] <span class="error">&#91;&lt;ffffffffa0a1fab0&gt;&#93;</span> filter_sync+0x80/0x600 <span class="error">&#91;obdfilter&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266457] <span class="error">&#91;&lt;ffffffffa09e039f&gt;&#93;</span> ost_blocking_ast+0x29f/0xa30 <span class="error">&#91;ost&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266485] <span class="error">&#91;&lt;ffffffffa06a36d6&gt;&#93;</span> ldlm_cancel_callback+0x56/0xe0 <span class="error">&#91;ptlrpc&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266504] <span class="error">&#91;&lt;ffffffffa06a37ac&gt;&#93;</span> ldlm_lock_cancel+0x4c/0x190 <span class="error">&#91;ptlrpc&#93;</span><br/> Oct 21 12:02:13 lus08-oss2 kernel: [ 3406.266528] <span class="error">&#91;&lt;ffffffffa06c3dcf&gt;&#93;</span> ldlm_request_cancel+0x13f/0x380 <span class="error">&#91;ptlrpc&#93;</span></p> <p>I asked them to turn down the oss threads to try to reduce contention on the disks and network, but that didn't seem to help. Let me know if there are any other logs you need. </p> SLES kernel on debian LU-2218

lots of small IO causes OST deadlock

Bug Minor Resolved Cannot Reproduce Niu Yawei Kit Westneat Mon, 22 Oct 2012 05:42:17 +0000 Fri, 9 May 2014 18:07:44 +0000 Fri, 9 May 2014 18:07:44 +0000 Lustre 1.8.8 0 7 <p>Niu</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>peter</p> <p>Here is additional information from Sanger:</p> <p>The pipeline runs 200-1500 concurrent jobs; each job creates its own<br/> working directory on lustre and processes ~2000 files within the<br/> directory. The file IO is all small; each job is typically opening a<br/> file, writing a couple of lines, closing the file. The process is then<br/> repeated on another file.</p> <p>There does not appear to be any concurrent write access to any of the<br/> files. At the end of the job, the working directory is deleted.</p> <p>All of these IO is happening on un-striped files &amp; directories.</p> <p>One thing we also noticed; the user concerned was at 98% of their inode<br/> quota, and it is entirely possible that they went over quota at some point.</p> <p>Unfortunately, their pipeline does not trap errors; it simply re-tries<br/> job if they fail. I have looked through the logs on all of the clients,<br/> and none of them have logged any of the lustre -122 errors that we would<br/> expect to see in an over-quota situation, but maybe the over-quota<br/> errors never makes it back from the deadlocked OSS.</p> <p>We have upped the user's quota, and re-activated the pipeline, so we<br/> should know in the next 24 hours whether this was the cause or not.</p> <p>Hi,</p> <p>We can reproduce the deadlock even when the user is well within quota.</p> <p>I've looked more closely at the logs and it looks like there is some kind of deadlock between 3 different functions:</p> <p>mutex_lock <br/> rwsem_down_failed_common<br/> start_this_handle</p> <p>They all seem to be called by either filter_destroy or ldlm_cancel_callback originally. </p> <p>Are there any debug flags that might help get information?</p> <p>Hi, Kit</p> <p>I checked the log, there are indeed many service threads are waiting for starting journal, and somes are waiting on the dqptr_sem, but it seems not necessary be a deadlock. I didn't see lock ordering issues from code so far (there was a deadlock problem, but it's already fixed in <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a>).</p> <p>Could you try to get the full stacktrace (on the hang OST) by sysrq? I think that could be helpful. Thanks.</p> <p>Hi, Yangshen</p> <p>I saw that ext4-back-dquot-to-rhel54.patch is removed by ba5dd769f66194a80920cf93d6014c78729efaae (<a href="https://jira.whamcloud.com/browse/LU-674" title="Update RHEL5 kernel to 2.6.18-274.3.1.el5 security update" class="issue-link" data-issue-key="LU-674"><del>LU-674</del></a>), won't it cause deadlock? (see b23235 530ddd7935bfbea6a9714c69d44da61f5efc098f).</p> <p>btw, Kit, what's the exact kernel version on OST?</p> <p>SLES kernel may different than RHEL. So the patches cannot apply directly. </p> <p>The kernel is version 2.6.32.59-sles-lustre-1.8.8wc1. I tried to see if there were any ext4 bugs fixed in more recent 2.6.32.y kernels, but I couldn't find any. We'll try to get a full stack trace the next time this occurred. We tried this time, but as you can see the first part was not saved for some reason.</p> <p>We have been running the same workload on one of our 2.2 file-systems for the past few days, and we not able to trigger the problem.</p> <p>Our plan will be to upgrade the problematic file-system to 2.X. I am happy for you to reduce the priority of the ticket in the meantime.<br/> Thanks,</p> <p>Guy</p> <p>Thank you, Guy. sles kernel isn't officially supported by 1.8.8, so I'm not sure if there is potential deadlock of journal lock with qptr_sem. Anyway, glad to hear you resolved the problem by upgrading.</p> Development Rank 1|hzvaqv: Rank (Obsolete) 5276 Severity