https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6898 Lustre <p><b>On the client we see errors</b></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[1437673998.848774] LustreError: 11-0: nbp8-MDT0000-mdc-ffff8806cb247400: Communicating with 10.151.27.60@o2ib, operation obd_ping failed with -107.^M [1437673998.860774] Lustre: nbp8-MDT0000-mdc-ffff8806cb247400: Connection to nbp8-MDT0000 (at 10.151.27.60@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete^M [1437673998.880773] LustreError: 167-0: nbp8-MDT0000-mdc-ffff8806cb247400: This client was evicted by nbp8-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail.^M [1437673998.916773] LustreError: 81375:0:(ldlm_resource.c:809:ldlm_resource_complain()) nbp8-MDT0000-mdc-ffff8806cb247400: namespace resource [0x360375393:0xe66d:0x0].0 (ffff8fc07bee8a80) refcount nonzero (1) after lock cleanup; forcing cleanup.^M [1437673998.940772] LustreError: 81375:0:(ldlm_resource.c:809:ldlm_resource_complain()) Skipped 2587 previous similar messages^M [1437673998.952772] LustreError: 81375:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x360375393:0xe66d:0x0].0 (ffff8fc07bee8a80) refcount = 2^M [1437673998.952772] LustreError: 81375:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order):^M [1437673998.952772] LustreError: 81375:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: nbp8-MDT0000-mdc-ffff8806cb247400 lock: ffff8fc0fa26dbc0/0x7f099458bb92bf52 lrc: 2/0,0 mode: PR/PR res: [0x360375393:0xe66d:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x12e400000000 nid: local remote: 0x551d423294fa4bce expref: -99 pid: 46426 timeout: 0 lvb_type: 3^M [1437673998.952772] LustreError: 81375:0:(ldlm_resource.c:1454:ldlm_resource_dump()) Skipped 3648 previous similar messages^M [1437673998.952772] LustreError: 81375:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x3603755cc:0x6454:0x0].0 (ffff8b075a9a8bc0) refcount = 2^M </pre> </div></div> <p><b>Server</b></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jul 23 10:53:08 nbp8-mds1 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 226s: evicting client at 10.151.63.50@o2ib ns: mdt-nbp8-MDT0000_UUID lock: ffff882a2c6794c0/0x551d4232f4dfbb5e lrc: 3/0,0 mode: PR/PR res: [0x4976d01:0xe2d4a4f3:0x0].0 bits 0x13 rrc: 848 type: IBT flags: 0x60200000000020 nid: 10.151.63.50@o2ib remote: 0x7f099458bb9c07c4 expref: 9 pid: 9672 timeout: 8029699438 lvb_type: 0 Jul 23 10:53:09 nbp8-mds1 kernel: LNet: 5828:0:(lib-move.c:865:lnet_post_send_locked()) Aborting message <span class="code-keyword">for</span> 12345-10.151.12.174@o2ib: LNetM[DE]Unlink() already called on the MD/ME. Jul 23 10:53:09 nbp8-mds1 kernel: LNet: 5828:0:(lib-move.c:865:lnet_post_send_locked()) Skipped 41 previous similar messages Jul 23 10:53:39 nbp8-mds1 kernel: format at ldlm_pool.c:628:ldlm_pool_recalc doesn't end in newline </pre> </div></div> <p>On the client all ldlm_bl threads are stuck </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">0xffff8a0739b06080 21185 2 1 711 R 0xffff8a0739b066f0 ldlm_bl_110^M [&lt;ffffffff814760e8&gt;] _raw_spin_unlock_irqrestore+0x8/0x10^M [&lt;ffffffffa0d7a807&gt;] osc_page_delete+0xe7/0x360 [osc]^M [&lt;ffffffffa0ad14d5&gt;] cl_page_delete0+0xc5/0x4e0 [obdclass]^M [&lt;ffffffffa0ad192a&gt;] cl_page_delete+0x3a/0x120 [obdclass]^M [&lt;ffffffffa0ee16a6&gt;] ll_invalidatepage+0x96/0x160 [lustre]^M [&lt;ffffffffa0ef314d&gt;] vvp_page_discard+0x8d/0x120 [lustre]^M [&lt;ffffffffa0acda58&gt;] cl_page_invoid+0x78/0x170 [obdclass]^M [&lt;ffffffffa0ad490c&gt;] discard_cb+0xbc/0x1e0 [obdclass]^M [&lt;ffffffffa0ad2467&gt;] cl_page_gang_lookup+0x1f7/0x3f0 [obdclass]^M [&lt;ffffffffa0ad471a&gt;] cl_lock_discard_pages+0xfa/0x1d0 [obdclass]^M [&lt;ffffffffa0d7c0d2&gt;] osc_lock_flush+0xf2/0x260 [osc]^M [&lt;ffffffffa0d7c339&gt;] osc_lock_cancel+0xf9/0x1e0 [osc]^M [&lt;ffffffffa0ad2bd5&gt;] cl_lock_cancel0+0x65/0x150 [obdclass]^M [&lt;ffffffffa0ad394b&gt;] cl_lock_cancel+0x14b/0x150 [obdclass]^M [&lt;ffffffffa0d7cc1d&gt;] osc_lock_blocking+0x5d/0xf0 [osc]^M [&lt;ffffffffa0d7dff9&gt;] osc_dlm_blocking_ast0+0xf9/0x210 [osc]^M [&lt;ffffffffa0d7e15c&gt;] osc_ldlm_blocking_ast+0x4c/0x100 [osc]^M [&lt;ffffffffa0be4eef&gt;] ldlm_cancel_callback+0x5f/0x180 [ptlrpc]^M [&lt;ffffffffa0bf380f&gt;] ldlm_cli_cancel_local+0x7f/0x480 [ptlrpc]^M [&lt;ffffffffa0bf6b82&gt;] ldlm_cli_cancel_list_local+0xf2/0x290 [ptlrpc]^M [&lt;ffffffffa0bfba07&gt;] ldlm_bl_thread_main+0xf7/0x450 [ptlrpc]^M [&lt;ffffffff81083ae6&gt;] kthread+0x96/0xa0^M [&lt;ffffffff8147f164&gt;] kernel_thread_helper+0x4/0x10^M </pre> </div></div> <p>These events will cause the MDS IO to stop for a few minutes. </p> CENTOS6 Lustre2.5.3 Server MOFED2.4<br/> SLES11 Lustre2.5.3 Client MOFED3.0 LU-6898

ldlm_resource_dump()) Granted locks (in reverse order)

Bug Major Resolved Cannot Reproduce Jinshan Xiong Mahmoud Hanafi Fri, 24 Jul 2015 01:41:55 +0000 Fri, 29 Apr 2016 17:58:44 +0000 Thu, 10 Mar 2016 22:02:17 +0000 Lustre 2.5.3 0 13 <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>the log shows that a client hadn't finished a lock cancellation while MDT thought that the client is dead and evicted it. </p> <p>why was this causing a pause on the MDS? <br/> FYI, the client is a 1024 core sharemem host. 100's of ldlm_bl threads are all stuck at osc_page_delete.</p> <p>Do you have log from the MDS?</p> <p>There isn't much more on the MDS logs other than the "lock callback timer expired" I tried to get a lustre debug dump but wan't able to capture it quickly enough. </p> <p>I think it could be that client's too busy cancelling locks but still miss the lock callback timeout, can you try <a href="http://review.whamcloud.com/#/c/14342/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14342/</a> and <a href="http://review.whamcloud.com/#/c/12603" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12603</a> on the client node?</p> <p>I need a b2_5 port of #12603. I have a conflict in lustre/ldlm/ldlm_request.c.</p> <p>here <a href="http://review.whamcloud.com/15800" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15800</a> is the b2_5 port of #12603</p> <p>Thank you , Zhenyu.</p> <p>We tried the patch and it didn't help. Running a 312 cpu ior job and canceling the ior run would cause all the ldlm treads to lockup in _raw_spin_unlock_irqrestore. Uploading debug logs</p> <p>debug.out.withpatch.ofed3.5.2.1438633632.bz2<br/> debug.out.withpatch.mofed.1438631502.bz2<br/> debug.out.withpatch.mofed.secondrun.1438631826.bz2</p> <p>Those threads were not stuck but simply busy. Also the eviction happened on MDC. How many threads did you notice on this state?</p> <p>I had a 312 CPU job so they or at least that many threads in that state. The threads never finish and cause the client to get evicted.</p> <p>please update this case.</p> <p>I will work on this issue.</p> <p>There are 128 ldlm block threads and all of them are busy at discarding pages. There was probably a long queue out there and it would take long waiting time before this blocking AST from MDT gets handled.</p> <p>Based on the reality that this node has 1024 cores, I will increase the number of ldlm threads and see how it goes. At present there is a kernel module parameter ldlm_num_threads but it has a hard limit as 128. Nobody could predict that there exists such fat node.</p> <p>If it doesn't help by increasing number dlm threads, I will make blocking callback of inodebits lock to have higher priority.</p> <p>We don't see this issue in 2.4.3 and can reproduce very easily in 2.5.3. So the behavior has changed. Any patches that may have caused this change?</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15960</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6898" title="ldlm_resource_dump()) Granted locks (in reverse order)" class="issue-link" data-issue-key="LU-6898"><del>LU-6898</del></a> ldlm: increase LDLM_NTHRS_MAX of ldlm threads<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 6ef32d9684b78d04bf67b936c28992a154f14345</p> <p>I will check it. Meanwhile, if you have hardware resource, you can figure it out by 'git bisect'.</p> <p>We see this same issue on compute nodes with lot of memory (1.5TB, 32 CPU's) (2.6.32-504.12.2.el6.x86_64, MOFED 2.3 and lustre 2.5.3.90)</p> <p>patch: <a href="http://review.whamcloud.com/15960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15960</a> helped some. The client will eventually worked through all the waiting ldlm_bl threads but it takes time. Where in 2.4.3 this was never the case. So still not fit for production on a large cpu node.</p> <p>bt during the wait.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[0]kdb&gt; btp 31094 Stack traceback <span class="code-keyword">for</span> pid 31094 0xffff8ac79e562540 31094 2 1 753 R 0xffff8ac79e562bb0 ldlm_bl_169 [&lt;ffffffff814760e8&gt;] _raw_spin_unlock_irqrestore+0x8/0x10 [&lt;ffffffffa0d08817&gt;] osc_page_delete+0xe7/0x360 [osc] [&lt;ffffffffa0c2f615&gt;] cl_page_delete0+0xc5/0x4e0 [obdclass] [&lt;ffffffffa0c2fa6a&gt;] cl_page_delete+0x3a/0x120 [obdclass] [&lt;ffffffffa1056776&gt;] ll_invalidatepage+0x96/0x160 [lustre] [&lt;ffffffffa106821d&gt;] vvp_page_discard+0x8d/0x120 [lustre] [&lt;ffffffffa0c2ba68&gt;] cl_page_invoid+0x78/0x170 [obdclass] [&lt;ffffffffa0c32d0c&gt;] discard_cb+0xbc/0x1e0 [obdclass] [&lt;ffffffffa0c305a7&gt;] cl_page_gang_lookup+0x1f7/0x3f0 [obdclass] [&lt;ffffffffa0c32b1a&gt;] cl_lock_discard_pages+0xfa/0x1d0 [obdclass] [&lt;ffffffffa0d0a0f2&gt;] osc_lock_flush+0xf2/0x260 [osc] [&lt;ffffffffa0d0a359&gt;] osc_lock_cancel+0xf9/0x1e0 [osc] [&lt;ffffffffa0c30fd5&gt;] cl_lock_cancel0+0x65/0x150 [obdclass] [&lt;ffffffffa0c31d4b&gt;] cl_lock_cancel+0x14b/0x150 [obdclass] [&lt;ffffffffa0d0ac3d&gt;] osc_lock_blocking+0x5d/0xf0 [osc] [&lt;ffffffffa0d0c019&gt;] osc_dlm_blocking_ast0+0xf9/0x210 [osc] [&lt;ffffffffa0d0c17c&gt;] osc_ldlm_blocking_ast+0x4c/0x100 [osc] [&lt;ffffffffa0ec7670&gt;] ldlm_handle_bl_callback+0xc0/0x420 [ptlrpc] [&lt;ffffffffa0ec7bd1&gt;] ldlm_bl_thread_main+0x201/0x450 [ptlrpc] [&lt;ffffffff81083ae6&gt;] kthread+0x96/0xa0 [0]more&gt; g Only <span class="code-quote">'q'</span> or <span class="code-quote">'Q'</span> are processed at more prompt, input ignored [&lt;ffffffff8147f164&gt;] kernel_thread_helper+0x4/0x10 r15 = 0xffff8fc79e187db0 r14 = 0xffff88a092987558 r13 = 0xffff8fc79e1875e0 r12 = 0xffffffff8147f31e bp = 0xffff88a092987558 bx = 0x0000000100033e68 r11 = 0xffffffffa1060bb0 r10 = 0x00000000000001e1 r9 = 0xffff88a736d76200 r8 = 0xffffffffa0d36e40 ax = 0xffffffffa0d36e58 cx = 0x0000000000000000 dx = 0xffffffffa0d36e58 si = 0x0000000000000282 di = 0x0000000000000282 orig_ax = 0xffffffffffffff01 ip = 0xffffffff814760e8 cs = 0x0000000000000010 </pre> </div></div> <p>Number of lock threads</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[mhanafi@endeavour2:~]$ ps -ef |grep ldlm_bl | wc -l 1295 [mhanafi@endeavour2:~]$ ps -ef |grep ldlm_cb | wc -l 438 </pre> </div></div> <p>Hi Mahmoud,</p> <p>Do you have a rough idea what type of locks(read, or write) are being canceled? For read lock, clients will have to do more work to check if the pages being destroyed are covered by other lock.</p> <p>hmm.. this is write lock.</p> <p>What's the spinlock at 'osc_page_delete+0xe7/0x360'? it looks like this lock is being highly contended.</p> <p>Which commit or tag of 2.5.3 (ie b2_5) would be a good starting point to establish the 'git bisect good' for this problem as suggested in <br/> <a href="https://jira.hpdd.intel.com/browse/LU-6898?focusedCommentId=123923&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-123923" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-6898?focusedCommentId=123923&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-123923</a></p> <p>what was the exact version number of 2.4 that was running okay on the node?</p> <p>Our version of 2.4 is based on 2.4.3 with some extra patches. You can see our git repo at<br/> <a href="https://github.com/jlan/lustre-nas/tree/nas-2.4.3" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/tree/nas-2.4.3</a></p> <p>The system in question is running a build at tag 2.4.3-12nasC, which corresponding to this:<br/> 8d46eeb - <a href="https://jira.whamcloud.com/browse/LU-4604" title="improve LFSCK async RPCs control" class="issue-link" data-issue-key="LU-4604"><del>LU-4604</del></a> obdclass: handle ldt_device_nr/ldt_linkage properly<br/> which was committed to nas-2.4.3 on Dec 23, 2014.</p> <p>I didn't find that area of code changed much from tag v2_4_3 to v2_5_3(I checked the extra patches applied to your branch as well).</p> <p>From the stack trace, it looks like all canceling threads are contended at LRU list lock at osc_lru_del(), which you can verify it by running the reproduction program with 'perf top' for example. But the thing is this area of code haven't changed that much so probably there exists a hidden issue, therefore I would like to hold the effort to create a patch now.</p> <p>There are roughly around 800 commits between v2_4_3 and v2_5_3. We can probably identify the problematic patch(es) in less than 10 iterations.</p> <p>b2_5 branch branched out at roughly<br/> "<a href="https://jira.whamcloud.com/browse/LU-1617" title="have git ignore generated files via .gitignore" class="issue-link" data-issue-key="LU-1617"><del>LU-1617</del></a> build: ignore automatically generated files,"<br/> which is close to v2_4_0 (<a href="https://jira.whamcloud.com/browse/LU-2784" title="Provide RPMs for automatic installation on Intel(R) Xeon Phi(TM) card" class="issue-link" data-issue-key="LU-2784"><del>LU-2784</del></a>).</p> <p>We tested a branch of <a href="https://jira.whamcloud.com/browse/LU-2784" title="Provide RPMs for automatic installation on Intel(R) Xeon Phi(TM) card" class="issue-link" data-issue-key="LU-2784"><del>LU-2784</del></a> from b2_5 tree and (YEAH!!!) it did not show the problem. The branch from v2_4_52 did show the problem!</p> <p>Our dedicated time on our big system ran out. We will continue tomorrow (6pm - 8pm) to narrow it down. If you can see the culprit by examining the code that would be even great!</p> <p>We have it! It is commit c8fd9c3 "<a href="https://jira.whamcloud.com/browse/LU-2850" title="Client support for upstream 3.8 kernel" class="issue-link" data-issue-key="LU-2850"><del>LU-2850</del></a> kernel: 3.8 upstream kills daemonize()" that caused the ldlm lock threads taking up 100% cpu time and not going away on large CPU-size systems!</p> <p>We tested the commit a35113b - "<a href="https://jira.whamcloud.com/browse/LU-2901" title="Duplicate filename on the same ldiskfs directory on MDS" class="issue-link" data-issue-key="LU-2901"><del>LU-2901</del></a> mdt: duplicate link names in directory", the one immediately before c8fd9c3 and it did not show the problem.</p> <p>It is a new kernel feature from Linux kernel 3.8 that you brought in. Somewhere in your code need to adapt to this new feature.</p> <p>Nice detective work Jay! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>This is a huge patch - I'm looking at it.</p> <p>Is that because there are signals pending for ldlm lock canceling threads?</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16063" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16063</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6898" title="ldlm_resource_dump()) Granted locks (in reverse order)" class="issue-link" data-issue-key="LU-6898"><del>LU-6898</del></a> ldlm: block signal for dlm block thread<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 3e676c6dd7d279fb846a20fd6750c45f14d617d6</p> <p>patch 16063 is to verify the idea by blocking signals of ldlm blocking threads. If it can fix the problem, I'm going to check updates of commit c8fd9c3 and decide which threads will need signals to be blocked.</p> <p>We do not have the system to test until tomorrow.</p> <p>BTW, I looked at the task_struct of one of the ldlm thread and saw its sigpending struct as<br/> below:</p> <p> shared_pending = {<br/> list = </p> { next = 0xffff8d87bb5ce160, prev = 0xffff8d87bb5ce160 } <p>, <br/> signal = {<br/> sig = </p> {18446618212836368752} <p> }<br/> }, </p> <p>The hex value of 18446618212836368752 is 0xffff8d87bb5ce170<br/> Does this make sense?</p> <p>Any suggestion of testing if our problem still exhibits with patch 16063 tomorrow?</p> <p>we should check the sigset task_struct::blocked of the thread in question.</p> <p>Most likely this is the problem. I will take further look if patch 16063 is not working.</p> <p>Jishan. Patch 16063 did not help. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>We have another block time reserved for Friday 4-6pm. Please advise if we should keep that reservation. A block time at Friday afternoon would prohibit jobs that needs more than 2 days to start.</p> <p>Do you still own the test node? It would be helpful to get a coredump and upload to our ftp site so that I can take a further look.</p> <p>Yes, please keep the reservation.</p> <p>I found a problem with the patch. I am updating it now.</p> <p>Taking a vmcore of a system of this size takes times. I am looking for your next patch. If that still fails we then will take a kdump.</p> <p>Also, "please keep the reservation" do you meant the block time now of the block time on Friday?</p> <p>That was an answer to the question 'Please advise if we should keep that reservation.'</p> <p>Please take a look at the btall.gz we uploaded on July 27, it was a backtrace of all kernel threads from a machine with 1024 cores and 4TB.</p> <p>We still have the vmcore of that incident.<br/> Checking the blocked signal in the vmcore, I found:<br/> 1) all kernel threads of "ldlm_bl_***" that with stack trace like this do NOT have blocked sigset:</p> <p>#16 <span class="error">&#91;ffff8fa79de1b9f8&#93;</span> osc_page_delete at ffffffffa0d7a807 <span class="error">&#91;osc&#93;</span><br/> #17 <span class="error">&#91;ffff8fa79de1ba28&#93;</span> cl_page_delete0 at ffffffffa0ad14d5 <span class="error">&#91;obdclass&#93;</span><br/> #18 <span class="error">&#91;ffff8fa79de1ba68&#93;</span> cl_page_delete at ffffffffa0ad192a <span class="error">&#91;obdclass&#93;</span><br/> #19 <span class="error">&#91;ffff8fa79de1ba88&#93;</span> ll_invalidatepage at ffffffffa0ee16a6 <span class="error">&#91;lustre&#93;</span><br/> #20 <span class="error">&#91;ffff8fa79de1bab8&#93;</span> vvp_page_discard at ffffffffa0ef314d <span class="error">&#91;lustre&#93;</span><br/> #21 <span class="error">&#91;ffff8fa79de1bad8&#93;</span> cl_page_invoid at ffffffffa0acda58 <span class="error">&#91;obdclass&#93;</span><br/> #22 <span class="error">&#91;ffff8fa79de1bb18&#93;</span> check_and_discard_cb at ffffffffa0ad5dac <span class="error">&#91;obdclass&#93;</span><br/> #23 <span class="error">&#91;ffff8fa79de1bb58&#93;</span> cl_page_gang_lookup at ffffffffa0ad2467 <span class="error">&#91;obdclass&#93;</span><br/> #24 <span class="error">&#91;ffff8fa79de1bbf8&#93;</span> cl_lock_discard_pages at ffffffffa0ad471a <span class="error">&#91;obdclass&#93;</span><br/> #25 <span class="error">&#91;ffff8fa79de1bc38&#93;</span> osc_lock_flush at ffffffffa0d7c0d2 <span class="error">&#91;osc&#93;</span><br/> #26 <span class="error">&#91;ffff8fa79de1bc88&#93;</span> osc_lock_cancel at ffffffffa0d7c339 <span class="error">&#91;osc&#93;</span><br/> #27 <span class="error">&#91;ffff8fa79de1bcc8&#93;</span> cl_lock_cancel0 at ffffffffa0ad2bd5 <span class="error">&#91;obdclass&#93;</span><br/> #28 <span class="error">&#91;ffff8fa79de1bce8&#93;</span> cl_lock_cancel at ffffffffa0ad394b <span class="error">&#91;obdclass&#93;</span><br/> #29 <span class="error">&#91;ffff8fa79de1bd08&#93;</span> osc_lock_blocking at ffffffffa0d7cc1d <span class="error">&#91;osc&#93;</span><br/> #30 <span class="error">&#91;ffff8fa79de1bd38&#93;</span> osc_dlm_blocking_ast0 at ffffffffa0d7dff9 <span class="error">&#91;osc&#93;</span><br/> #31 <span class="error">&#91;ffff8fa79de1bd78&#93;</span> osc_ldlm_blocking_ast at ffffffffa0d7e15c <span class="error">&#91;osc&#93;</span><br/> #32 <span class="error">&#91;ffff8fa79de1bdb8&#93;</span> ldlm_cancel_callback at ffffffffa0be4eef <span class="error">&#91;ptlrpc&#93;</span><br/> #33 <span class="error">&#91;ffff8fa79de1bdc8&#93;</span> ldlm_cli_cancel_local at ffffffffa0bf380f <span class="error">&#91;ptlrpc&#93;</span><br/> #34 <span class="error">&#91;ffff8fa79de1bde8&#93;</span> ldlm_cli_cancel_list_local at ffffffffa0bf6b82 <span class="error">&#91;ptlrpc&#93;</span><br/> #35 <span class="error">&#91;ffff8fa79de1be48&#93;</span> ldlm_bl_thread_main at ffffffffa0bfba07 <span class="error">&#91;ptlrpc&#93;</span><br/> #36 <span class="error">&#91;ffff8fa79de1bee8&#93;</span> kthread at ffffffff81083ae6<br/> #37 <span class="error">&#91;ffff8fa79de1bf48&#93;</span> kernel_thread_helper at ffffffff8147f164</p> <p>2) all kernel threads of "ldlm_cb*<b>_</b>**" with stack trace like this:<br/> PID: 4677 TASK: ffff8834baff84c0 CPU: 18 COMMAND: "ldlm_cb00_008"<br/> #0 <span class="error">&#91;ffff88366f143cb0&#93;</span> schedule at ffffffff81473d9b<br/> #1 <span class="error">&#91;ffff88366f143df8&#93;</span> ptlrpc_wait_event at ffffffffa0c25445 <span class="error">&#91;ptlrpc&#93;</span><br/> #2 <span class="error">&#91;ffff88366f143e78&#93;</span> ptlrpc_main at ffffffffa0c2e91b <span class="error">&#91;ptlrpc&#93;</span><br/> #3 <span class="error">&#91;ffff88366f143ee8&#93;</span> kthread at ffffffff81083ae6<br/> #4 <span class="error">&#91;ffff88366f143f48&#93;</span> kernel_thread_helper at ffffffff8147f164<br/> DOES have a blocked sigset:<br/> blocked = {<br/> sig = </p> {18446744073709551615}<br/> }, <br/> <br/> 3) all kernel threads of "ptlrpc_hr*<b>_</b>**" with stack trace of:<br/> #0 <span class="error">&#91;ffff88679ecb9ce0&#93;</span> schedule at ffffffff81473d9b<br/> #1 <span class="error">&#91;ffff88679ecb9d88&#93;</span> cfs_cpt_bind at ffffffffa098a0bd <span class="error">&#91;libcfs&#93;</span><br/> #2 <span class="error">&#91;ffff88679ecb9e28&#93;</span> ptlrpc_hr_main at ffffffffa0c28ad7 <span class="error">&#91;ptlrpc&#93;</span><br/> #3 <span class="error">&#91;ffff88679ecb9ee8&#93;</span> kthread at ffffffff81083ae6<br/> #4 <span class="error">&#91;ffff88679ecb9f48&#93;</span> kernel_thread_helper at ffffffff8147f164<br/> DOES have a blocked sigset:<br/> blocked = {<br/> sig = {18446744073709551615} <p> }, </p> <p>3) all kernel threads of "kiblnd_sd_<cite>_</cite>" with stack trace of<br/> PID: 25939 TASK: ffff88e768552040 CPU: 24 COMMAND: "kiblnd_sd_00_00"<br/> #0 <span class="error">&#91;ffff88e768555cb0&#93;</span> schedule at ffffffff81473d9b<br/> #1 <span class="error">&#91;ffff88e768555df8&#93;</span> kiblnd_scheduler at ffffffffa0ba52a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> #2 <span class="error">&#91;ffff88e768555ee8&#93;</span> kthread at ffffffff81083ae6<br/> #3 <span class="error">&#91;ffff88e768555f48&#93;</span> kernel_thread_helper at ffffffff8147f164<br/> DOES have a blocked sigset:<br/> blocked = {<br/> sig = </p> {18446744073709551615}<br/> }, <br/> <br/> 4) all kernel threads of "ptlrpcd_????" with stack trace of<br/> PID: 27092 TASK: ffff8f079ef16340 CPU: 1019 COMMAND: "ptlrpcd_1023"<br/> #0 <span class="error">&#91;ffff8f079ef19c30&#93;</span> schedule at ffffffff81473d9b<br/> #1 <span class="error">&#91;ffff8f079ef19d78&#93;</span> schedule_timeout at ffffffff81474550<br/> #2 <span class="error">&#91;ffff8f079ef19e08&#93;</span> ptlrpcd at ffffffffa0c3d7c5 <span class="error">&#91;ptlrpc&#93;</span><br/> #3 <span class="error">&#91;ffff8f079ef19ee8&#93;</span> kthread at ffffffff81083ae6<br/> #4 <span class="error">&#91;ffff8f079ef19f48&#93;</span> kernel_thread_helper at ffffffff8147f164<br/> DOES have a blocked sigset:<br/> blocked = {<br/> sig = {18446744073709551615} <p> }, </p> <p>It seems that all lustre kernel threads except #1 category ("ldlm_bl_???") have a blocked sigset. The content of the sigset all the same.</p> <p>Hi Jay,</p> <p>I realized that the new patch set 16063 won't fix the problem either. I have no idea how commit c8fd9c3 could cause this problem. I will do furhther investigation and will update this ticket if I find something new.</p> <p>Hi Jinshan,</p> <p>If you get the conclusion because of the comment I made a few hour before, please be advised that the blocked sigset data were from the vmcore dated July 27. I do not have a chance to test your new patch 16063 (patch set #2) yet.</p> <p>In that case, please try the 2nd patch anyway as I'm investigating the problem.</p> <p>Please take a coredump if the problem still exists so that we can do postmortem analysis.</p> <p>We reproduced the problem with patch at<br/> <a href="http://review.whamcloud.com/#/c/16063/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16063/</a></p> <p>We took a vmcore. Either Mahmoud or Herbert will<br/> encrypt and ftp to your server. Please note that the vmcore is restricted to US citizen.</p> <p>The git repo is at<br/> <a href="https://github.com/jlan/lustre-nas/tree/nas-2.5.3-LU6898" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/tree/nas-2.5.3-LU6898</a></p> <p>hmm.. I will take a look at coredump after it's uploaded.</p> <p>Do you have another chance to run a test again to make sure that the problem does not exist in commit a35113b? I can't figure out why commit c8fd9c3 could cause this problem otherwise I will escalate this ticket.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16165" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16165</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6898" title="ldlm_resource_dump()) Granted locks (in reverse order)" class="issue-link" data-issue-key="LU-6898"><del>LU-6898</del></a> ldlm: debug patch<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 7bcd94d94c915f6f3670f6355494daaa66662149</p> <p>I pushed patch 16165 to revert the changes to ldlm and ptlrpc in c8fd9c3. Please check if you can still see the problem by applying this patch.</p> <p>Patch 16165 can't be the final fix since daemonize is actually gone for new kernels like RHEL7 and SLES12. Jay Lan how are your producing this problem. I like to see if I can duplicate it.</p> <p>Obviously it isn't - it's a debug patch to identify the problem as the patch title says.</p> <p>Tommi Tervo also reported the problem at 12/Aug/15 6:21 AM. His system is a 1.5TB,32-cores.</p> <p>We can not reproduce the problem on small systems, but can easily reproduce on our 2TB,512-cores SGI UV2000 system.</p> <p>In Mahmoud's reproducer, the lustre fs uses stripe-count=10. His script submits PBS job to run mpiexec of 312 copies of IOR.</p> <p>Using 'top' to monitor the run. When you see IOR takes up the 'top' page, ctrl-C to kill the job. Then, if you see all ldlm_bl_xxx threads showing up in 'top' pages taking ~100% CPU time, the problem is reproduced. Sometimes it takes a couple of runs to reproduce the problem.</p> <p>The reproducer runs on single stripe file per process IOR. Not stripe count of 10.</p> <p>Jinshan,</p> <p>8 files of your patch caused conflicts.</p> <p>Could create a branch at commit c8fd9c3 and create your patch from there? Commit c8fd9c3 is where the problem starts.</p> <p>the patch is based on b2_5, please apply it to the top of b2_5.</p> <p>Is it possible to enable lock stat and collect perf(1) data so that we know if there are contention in the code?</p> <p>Have you found anything from the crash dump, Jinshan and Oleg?</p> <p>We found that the patch from Jinshan managed to set blocking callback just like it was supposed to, so there now should be no differences prior to the patch you mention.<br/> I also see that all ldlm_bl callbacks are having very uniform backtraces all of them culminating at released spinlock which is kind of strange to be a coincidence considering there are 128 of them, but I have no explanation for it yet.</p> <p>We are still thinking of the reasons, but meanwhile Jinshan is very interested in b2_5 + the patch he just reverted to be run on your system to see how it helps.</p> <p>After applying the debug patch to nas-2.5.3, we can not reproduce the problem.</p> <p>What feature requires commit c8fd9c3? Do we need this patch on sles11sp3?</p> <p>As far as I know, there is no new feature that requires commit c8fd9c3. It will just make lustre code more in line with kernel code.</p> <p>I will make further investigation on this issue. Please apply the debug patch in your release while we're investigating this issue.</p> <p>The ldlm_bl_xxx callbacks were the ones stalled the system. They all ran at ~100% cpu, with commit c8fd9c3.</p> <p>Hi Jinshan and Mahamoud,<br/> Recently our MDS servers are frequently stucked by some Lustre clients. The symptom is same as Mahmoud Hanafi reported in this track and perhaps <a href="https://jira.hpdd.intel.com/browse/LU-4572" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4572</a><br/> It happens in this sequence:<br/> 1. one client reports -107 connection error with a MDS server<br/> 2. it starts to release locks, a lot of ldlm_resource_dump found in syslog<br/> 3. During these period ldlm_bl threads take 100% CPU usage on the client, and mds reports "### lock expired XXX seconds" with this client. <br/> 4. Other clients find the MDS stuck.<br/> Since we have 7 Lustre instances, sometimes, one client will has 107 error with more than one MDS server, then this client would stuck more than one Lustre instance. <br/> Our current solution is manually 1) restart the client or 2) remount the MDT device. Do you have any updates or workaround on this problem?</p> <p>By the way, our Lustre version is 2.5.3 without any modification/patch. </p> <p>The back trace of ldlm_bl:<br/> &lt;6&gt;ldlm_bl_19 R running task 0 8801 2 0x00000000<br/> &lt;4&gt; ffff8802d24b3b30 ffffffff8129120a 000000000003ffff ffff880201c33a00<br/> &lt;4&gt; ffff8802d24b3b10 ffffffffa0527a12 ffff8802d24b3b30 ffffffffa0868809<br/> &lt;4&gt; 000000000000000e 000000000000000e ffff8802d24b3be0 ffffffffa052a687<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8129120a&gt;&#93;</span> ? radix_tree_gang_lookup+0x7a/0xf0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0527a12&gt;&#93;</span> ? cl_page_vmpage+0x82/0xb0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0868809&gt;&#93;</span> ? weigh_cb+0x19/0x50 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052a687&gt;&#93;</span> cl_page_gang_lookup+0x277/0x3f0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa08687f0&gt;&#93;</span> ? weigh_cb+0x0/0x50 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa08689b7&gt;&#93;</span> osc_ldlm_weigh_ast+0x177/0x400 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa084d05d&gt;&#93;</span> osc_cancel_for_recovery+0x5d/0xa0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa064011b&gt;&#93;</span> ldlm_cancel_lrur_policy+0xeb/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0643f2b&gt;&#93;</span> ldlm_prepare_lru_list+0x1cb/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0640030&gt;&#93;</span> ? ldlm_cancel_lrur_policy+0x0/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0644344&gt;&#93;</span> ldlm_cancel_lru_local+0x24/0x40 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0644d02&gt;&#93;</span> ldlm_cli_cancel+0x122/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0869877&gt;&#93;</span> osc_ldlm_blocking_ast+0x1b7/0x350 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8152a61d&gt;&#93;</span> ? wait_for_completion+0x1d/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06483c0&gt;&#93;</span> ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06488f1&gt;&#93;</span> ldlm_bl_thread_main+0x261/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81064b90&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0648690&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109e66e&gt;&#93;</span> kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109e5d0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> &lt;6&gt;ldlm_bl_19 R running task 0 8802 2 0x00000000<br/> &lt;4&gt; ffff88027f49bbb0 0000000000000046 000000000003ffff ffff880391a08000<br/> &lt;4&gt; ffff88027f49bb10 ffffffffa0527a12 ffff88027f49bb30 ffffffffa0868809<br/> &lt;4&gt; 000000000000000e 000000000000000e ffff88082aa4f098 ffff88027f49bfd8<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0527a12&gt;&#93;</span> ? cl_page_vmpage+0x82/0xb0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0868809&gt;&#93;</span> ? weigh_cb+0x19/0x50 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa08687f0&gt;&#93;</span> ? weigh_cb+0x0/0x50 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8106c85a&gt;&#93;</span> __cond_resched+0x2a/0x40<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8152a3d0&gt;&#93;</span> _cond_resched+0x30/0x40<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0868b25&gt;&#93;</span> osc_ldlm_weigh_ast+0x2e5/0x400 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa084d05d&gt;&#93;</span> osc_cancel_for_recovery+0x5d/0xa0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa064011b&gt;&#93;</span> ldlm_cancel_lrur_policy+0xeb/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0643f2b&gt;&#93;</span> ldlm_prepare_lru_list+0x1cb/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0640030&gt;&#93;</span> ? ldlm_cancel_lrur_policy+0x0/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0644344&gt;&#93;</span> ldlm_cancel_lru_local+0x24/0x40 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0644d02&gt;&#93;</span> ldlm_cli_cancel+0x122/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0869877&gt;&#93;</span> osc_ldlm_blocking_ast+0x1b7/0x350 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06483c0&gt;&#93;</span> ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06488f1&gt;&#93;</span> ldlm_bl_thread_main+0x261/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81064b90&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0648690&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109e66e&gt;&#93;</span> kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109e5d0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>The client syslog:<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 11-0: bes3fs-OST0051-osc-ffff881070edf000: Communicating with 192.168.50.165@tcp, operation obd_ping failed with -107.<br/> Jan 5 18:12:49 bwsjnws020 kernel: Lustre: pubfs-MDT0000-mdc-ffff88105ab8e400: Connection to pubfs-MDT0000 (at 192.168.50.67@tcp) was lost; in progress operations using this service will wait for recovery to complete<br/> Jan 5 18:12:49 bwsjnws020 kernel: Lustre: Skipped 3 previous similar messages<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: Skipped 6 previous similar messages<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 167-0: pubfs-MDT0000-mdc-ffff88105ab8e400: This client was evicted by pubfs-MDT0000; in progress operations using this service will fail.<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: Skipped 3 previous similar messages<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 1058:0:(ldlm_resource.c:809:ldlm_resource_complain()) pubfs-MDT0000-mdc-ffff88105ab8e400: namespace resource <span class="error">&#91;0x200007898:0x1:0x0&#93;</span>.0 (ffff8806de1651c0) refcount nonzero (2) after lock cleanup; forcing cleanup.<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 1058:0:(ldlm_resource.c:809:ldlm_resource_complain()) Skipped 7 previous similar messages<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 1058:0:(ldlm_resource.c:1448:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x200007898:0x1:0x0&#93;</span>.0 (ffff8806de1651c0) refcount = 3<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 1058:0:(ldlm_resource.c:1448:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x2000074d1:0x94e:0x0&#93;</span>.0 (ffff8801d79be340) refcount = 3<br/> Jan 5 18:12:49 bwsjnws020 kernel: Lustre: pubfs-MDT0000-mdc-ffff88105ab8e400: Connection restored to pubfs-MDT0000 (at 192.168.50.67@tcp)<br/> Jan 5 18:12:49 bwsjnws020 kernel: Lustre: Skipped 3 previous similar messages<br/> Jan 5 18:12:49 bwsjnws020 kernel: LustreError: 1059:0:(ldlm_resource.c:1448:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x1b995:0x0:0x0&#93;</span>.0 (ffff880721859700) refcount = 2</p> <p>The server log:<br/> Jan 5 18:23:27 smds kernel: Lustre: 10627:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:23:27 smds kernel: req@ffff8812ab48ec00 x1518974973017848/t0(0) o101-&gt;c82eb4b4-c8b9-07c2-e3fc-87539e7b2adf@192.168.54.29@tcp:0/0 lens 576/3384 e 6 to 0 dl 1451989412 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:23:27 smds kernel: Lustre: 10627:0:(service.c:1347:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages<br/> Jan 5 18:23:28 smds kernel: Lustre: 10819:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:23:28 smds kernel: req@ffff880650196400 x1513068049228408/t0(0) o101-&gt;5c631268-d0fc-8c0d-05f8-c3d7a7faae99@192.168.57.199@tcp:0/0 lens 576/3384 e 6 to 0 dl 1451989413 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:23:28 smds kernel: Lustre: 10819:0:(service.c:1347:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages<br/> Jan 5 18:23:34 smds kernel: Lustre: 10534:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:23:34 smds kernel: req@ffff88169949fc00 x1509295010536664/t0(0) o101-&gt;9dda6a2d-4137-995a-cad9-a28186cc8b73@192.168.57.44@tcp:0/0 lens 576/3384 e 6 to 0 dl 1451989419 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:23:34 smds kernel: Lustre: 10534:0:(service.c:1347:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message<br/> Jan 5 18:23:47 smds kernel: Lustre: 10819:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:23:47 smds kernel: req@ffff880213478800 x1520526088261116/t0(0) o101-&gt;fd06b39c-b81c-d983-43f8-89c8672bb709@202.122.33.204@tcp:0/0 lens 544/3384 e 6 to 0 dl 1451989432 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:24:03 smds kernel: Lustre: 10819:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:24:03 smds kernel: req@ffff880a487c0000 x1520538965607805/t0(0) o101-&gt;1a2f04d1-18eb-410e-f207-5222c830edf2@202.122.33.209@tcp:0/0 lens 544/3384 e 4 to 0 dl 1451989448 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:24:51 smds kernel: Lustre: 10819:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply<br/> Jan 5 18:24:51 smds kernel: req@ffff8803751db800 x1520526088262835/t0(0) o101-&gt;fd06b39c-b81c-d983-43f8-89c8672bb709@202.122.33.204@tcp:0/0 lens 544/3384 e 2 to 0 dl 1451989496 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:25:03 smds kernel: Lustre: 10554:0:(service.c:2039:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:90s); client may timeout. req@ffff8817e6235000 x1509380311299204/t0(0) o101-&gt;d0626c3e-a7a7-7421-ed73-e28aa9242043@192.168.61.234@tcp:0/0 lens 576/536 e 6 to 0 dl 1451989413 ref 1 fl Complete:/0/0 rc 0/0<br/> Jan 5 18:25:03 smds kernel: Lustre: 10554:0:(service.c:2039:ptlrpc_server_handle_request()) Skipped 12 previous similar messages<br/> Jan 05 18:33:01 smds ihep_heartbeat_log: the heplog system is collecting logs. @Jan 05 18:33:01<br/> Jan 5 18:34:59 smds kernel: Lustre: 10547:0:(service.c:1347:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/4), not sending early reply<br/> Jan 5 18:34:59 smds kernel: req@ffff8815354d3000 x1519342037432032/t0(0) o101-&gt;740c17eb-17dd-83c0-ad32-6236642ecf32@202.122.33.224@tcp:0/0 lens 576/3384 e 0 to 0 dl 1451990104 ref 2 fl Interpret:/0/0 rc 0/0<br/> Jan 5 18:35:04 smds kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 601s: evicting client at 192.168.85.14@tcp ns: mdt-scrafs-MDT0000_UUID lock: ffff880b43a1e1c0/0x24184c57e1317020 lrc: 3/0,0 mode: PR/PR res: <span class="error">&#91;0xa000001:0xb432f5:0x0&#93;</span>.0 bits 0x13 rrc: 560 type: IBT flags: 0x60200000000020 nid: 192.168.85.14@tcp remote: 0x42dffa022b197861 expref: 22 pid: 10741 timeout: 15444565589 lvb_type: 0<br/> Jan 5 18:35:04 smds kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) Skipped 4 previous similar messages<br/> Jan 5 18:35:04 smds kernel: LustreError: 10673:0:(ldlm_lockd.c:1335:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880bbf050800 ns: mdt-scrafs-MDT0000_UUID lock: ffff8807bae9db80/0x24184c57e1317090 lrc: 3/0,0 mode: PR/PR res: <span class="error">&#91;0xa000001:0xb432f5:0x0&#93;</span>.0 bits 0x13 rrc: 628 type: IBT flags: 0x50200400000020 nid: 192.168.85.14@tcp remote: 0x42dffa022b197868 expref: 7 pid: 10673 timeout: 0 lvb_type: 0</p> <p>Hi Lu,</p> <p>Have you noticed that there was heavy contention on some locks when this issue occurred? It looks like the OSC in question was in recovery state therefore it picked some read locks to cancel before recovery.</p> <p>Has there been any progress on fixing this issue?</p> <p>We haven't seen this issue. Close will reopen if needed.</p> Development Rank 1|hzxivz: Rank (Obsolete) 9223372036854775807 Severity