https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1107 Lustre <p>our customer is seeing what OSS is dumping the following Call traces occasionally.</p> <p>This is similar to <a href="https://jira.whamcloud.com/browse/LU-745" title="ost-pools test 23 hung" class="issue-link" data-issue-key="LU-745"><del>LU-745</del></a>, but this customer installed lustre-1.8.7 and <a href="https://jira.whamcloud.com/browse/LU-745" title="ost-pools test 23 hung" class="issue-link" data-issue-key="LU-745"><del>LU-745</del></a> should be fixed.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8002e024&gt;] __wake_up+0x38/0x4f Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88aba7f3&gt;] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff800a2dff&gt;] autoremove_wake_function+0x0/0x2e Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88b6590b&gt;] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88ba6194&gt;] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88b47d09&gt;] ost_brw_write+0x1c99/0x2480 [ost] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88881ac8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8884c8b0&gt;] target_committed_to_req+0x40/0x120 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff888860a8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88b4b09e&gt;] ost_handle+0x2bae/0x55b0 [ost] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff888956d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88895e35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88896dc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff88895e60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Feb 14 15:03:45 oss02 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 </pre> </div></div> <p>The another call trace is here</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff80046823&gt;] try_to_wake_up+0x27/0x484 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8028b1ca&gt;] __down_trylock+0x39/0x4e Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8006473d&gt;] __down_failed_trylock+0x35/0x3a Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8886c0c1&gt;] ldlm_pool_shrink+0x31/0xf0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8884a1e6&gt;] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8886d24c&gt;] ldlm_pools_shrink+0x15c/0x2f0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff80064614&gt;] __down_read+0x12/0x92 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8003f285&gt;] shrink_slab+0xdc/0x153 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff800ce4ce&gt;] zone_reclaim+0x235/0x2cd Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff800ca81e&gt;] __rmqueue+0x44/0xc6 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8000a939&gt;] get_page_from_freelist+0xbf/0x442 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8000f46f&gt;] __alloc_pages+0x78/0x308 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff80025e20&gt;] find_or_create_page+0x32/0x72 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88b98445&gt;] filter_get_page+0x35/0x70 [obdfilter] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88b9a68a&gt;] filter_preprw+0x14da/0x1e00 [obdfilter] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8876d121&gt;] LNetMDBind+0x301/0x450 [lnet] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff887d5d30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88b4500c&gt;] ost_brw_write+0xf9c/0x2480 [ost] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8876d121&gt;] LNetMDBind+0x301/0x450 [lnet] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88889c65&gt;] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88883fe5&gt;] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88883ef5&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff888840a8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88b4909e&gt;] ost_handle+0x2bae/0x55b0 [ost] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff887d5d30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8883e19a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88843168&gt;] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff888936d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88893e35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88894dc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff88893e60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Feb 15 11:10:47 oss03 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 </pre> </div></div> <p>any related to bugs?</p> LU-1107

OSS dumps Call traces occasionally

Bug Minor Resolved Cannot Reproduce Oleg Drokin Shuichi Ihara Wed, 15 Feb 2012 13:00:19 +0000 Mon, 29 May 2017 03:19:42 +0000 Mon, 29 May 2017 03:19:42 +0000 Lustre 1.8.7 0 5 <p>The two call traces are quite of different nature I would say.<br/> The first sample looks like some sort of journal deadlock. In that case it would have been very useful to have backtraces for all processes in the system. I checked the logs and the samples there are very limited.</p> <p>The other stacktrace looks like an outt of memory induced ldlm shrink problems, I think Johann battled quite a bit of those not so long ago.<br/> I propose you to separate these two reports into two different bugs.</p> <p>got backtrace when the problem happens.</p> <p>In addtion, there are some updates. During OSS is dumping the calltraces, the client can't access to servers, but after 30-60mins server recovered without reboot or other any actions.</p> <p>The customer hit same situation again today. They did sysrq-t to get backtrace. pleaes have a look at attached all oss log files.</p> <p>What is in the logs prior to the stack dumps? There should be a related error, indicating why the stack was dumped.</p> <p>hmm.. they hit same situation that client can't access to the servers, but not many call traces today. <br/> see messages_oss03 at Feb 16 13:12:15 and Feb 16 22:30:30, these are when the problme happened.</p> <p>this is client log. it was hanging when the problem happens. Is it related to <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>?</p> <p>OK, I want to focus jbd2 related problem on this ticket. Please change the title to "jbd2 deadlock" whatever reasonable title name.</p> <p>Today, we saw same jbd2 related call trace on an OSS, and got sysrq -t before umount/mount OSTs.</p> <p>After umount/mount OSTs without reboot, the problem was gone. But, In fact, the same problem happened again.</p> <p>the log files when the problem happened again at 02/22</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 22 06:02:18 oss01 kernel: Lustre: Service thread pid 18979 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 22 06:02:18 oss01 kernel: Pid: 18979, comm: ll_ost_io_78 Feb 22 06:02:18 oss01 kernel: Feb 22 06:02:18 oss01 kernel: Call Trace: Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff889f47f3&gt;] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff800a2dff&gt;] autoremove_wake_function+0x0/0x2e Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff88a9f90b&gt;] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff88ae0194&gt;] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff88a81d09&gt;] ost_brw_write+0x1c99/0x2480 [ost] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff887bbac8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff887868b0&gt;] target_committed_to_req+0x40/0x120 [ptlrpc] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff887c00a8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff88a8509e&gt;] ost_handle+0x2bae/0x55b0 [ost] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff88711d30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff8877a19a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] Feb 22 06:02:18 oss01 kernel: [&lt;ffffffff8877f168&gt;] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cf6d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cfe35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887d0dc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cfe60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 Feb 22 06:02:19 oss01 kernel: Feb 22 06:02:19 oss01 kernel: LustreError: dumping log to /tmp/lustre-log.1329858138.18979 Feb 22 06:02:19 oss01 kernel: Lustre: Service thread pid 18919 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 22 06:02:19 oss01 kernel: Pid: 18919, comm: ll_ost_io_18 Feb 22 06:02:19 oss01 kernel: Feb 22 06:02:19 oss01 kernel: Call Trace: Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80062ff2&gt;] thread_return+0x62/0xfe Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80046c6e&gt;] try_to_wake_up+0x472/0x484 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80064a0b&gt;] __down+0xc3/0xd8 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff800646c9&gt;] __down_failed+0x35/0x3a Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88a316a0&gt;] ldiskfs_destroy_inode+0x0/0x90 [ldiskfs] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887861e6&gt;] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887a9337&gt;] ldlm_pools_shrink+0x247/0x2f0 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80064614&gt;] __down_read+0x12/0x92 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff800223e9&gt;] __up_read+0x19/0x7f Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8003f279&gt;] shrink_slab+0xd0/0x153 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff800ce4ce&gt;] zone_reclaim+0x235/0x2cd Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80017ce1&gt;] cache_grow+0x360/0x3c7 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8000a939&gt;] get_page_from_freelist+0xbf/0x442 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8000f46f&gt;] __alloc_pages+0x78/0x308 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff80025e20&gt;] find_or_create_page+0x32/0x72 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88ad4445&gt;] filter_get_page+0x35/0x70 [obdfilter] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88ad668a&gt;] filter_preprw+0x14da/0x1e00 [obdfilter] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff886a9121&gt;] LNetMDBind+0x301/0x450 [lnet] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88711d30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88a8100c&gt;] ost_brw_write+0xf9c/0x2480 [ost] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887bbac8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887868b0&gt;] target_committed_to_req+0x40/0x120 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887bffe5&gt;] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887bfef5&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887c00a8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88a8509e&gt;] ost_handle+0x2bae/0x55b0 [ost] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff88711d30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8877a19a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8877f168&gt;] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cf6d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cfe35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887d0dc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff887cfe60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Feb 22 06:02:19 oss01 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 </pre> </div></div> <p>This second stacktrace is the culprit I guess. Attempting to allocate some mmemory got into ldlm to drop ldiskfs inode which might hold a journal or something and upsettign the other thread as a result.</p> <p>I think we had some patches about it that I am now trying to find.</p> <p>hm, on the server side, the ldlm pool shrinker is "just" supposed to decrease the SLV (see ldlm_srv_pool_shrink()), so i'm not sure to understand how it can end up in ldiskfs_destroy_inode() ... really odd.</p> <p>sorry, they did only apply patches to OSSs. we should apply the patches of <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a> to both OSSs and clients, right?</p> <p>Ihara, the patch from <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a> only helps on the client side, so i don't think it will make any difference with this bug which is an OSS issue.<br/> Could you please tell us what patches you applied on the OSS side?</p> <p>Johann,<br/> OK... they applied <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a> patches only to OSSs, but not to the clients. This was our miscommunication, sorry. let we try the same patch to clients in this time, just in case....</p> <p>The customer tested patch for <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a> on the client, but it didn't help. The problem seems to be easy replicate. Just copy the data the local filesystem to lustre with rsync command. (e.g. rsync -avHS --exclude=/lustre --exclude=/dev --exclude=/proc --exclude=/sys /* /lustre/)</p> <p>Please investigate.. The production system is not working well if one client runs rsync on the client... </p> <p>Ihara</p> <p>I am sorry if we have not been clear enough - we did not expect the patch from <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a> to help with this issue as it is a different kind of issue. What is blocking us moving forward is to get details from you about whether any patches have been applied to the vanilla release</p> <p>Regards</p> <p>Peter</p> <p>Peter,</p> <p>Yes, I agreed. Johann suggested the patch doesn't help. The customer did test just in case..<br/> Anyway, we only applied a patch (<a href="http://review.whamcloud.com/#change,1919" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1919</a>) to lustre-1.8.7-wc1 on this site.<br/> The reason of what we applied this patch is that we hit an issue on <a href="https://jira.whamcloud.com/browse/LU-967" title="OSS hangs due to heavey IO loads" class="issue-link" data-issue-key="LU-967"><del>LU-967</del></a> and <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a> at different sites. (Just in order to prevent same issue on this site)</p> <p>Oleg suggested the following changes and we tested on it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--- a/lustre/ldlm/ldlm_pool.c +++ b/lustre/ldlm/ldlm_pool.c @@ -1062,8 +1062,7 @@ static int ldlm_pools_shrink(ldlm_side_t client, int nr, int total = 0, cached = 0, nr_ns; struct ldlm_namespace *ns; - if (client == LDLM_NAMESPACE_CLIENT &amp;&amp; nr != 0 &amp;&amp; - !(gfp_mask &amp; __GFP_FS)) + if (nr != 0 &amp;&amp; !(gfp_mask &amp; __GFP_FS)) return -1; if (nr != 0) </pre> </div></div> <p>We didn't see same issue and same jbd2 calltrace on OSS and copy the files with rsync worked well.<br/> Howerver, we got another calltrace here during the same testing.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 29 16:41:17 oss03 kernel: Call Trace: Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff80046c1e&gt;] try_to_wake_up+0x422/0x484 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8028b1ca&gt;] __down_trylock+0x39/0x4e Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8006473d&gt;] __down_failed_trylock+0x35/0x3a Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887c20f4&gt;] ldlm_pool_shrink+0x64/0xf0 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887a01dc&gt;] .text.lock.ldlm_resource+0x73/0x87 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887c332f&gt;] ldlm_pools_shrink+0x23f/0x2d0 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff80064614&gt;] __down_read+0x12/0x92 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff800223e9&gt;] __up_read+0x19/0x7f Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8003f285&gt;] shrink_slab+0xdc/0x153 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff800ce4ce&gt;] zone_reclaim+0x235/0x2cd Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8000985a&gt;] __d_lookup+0xb0/0xff Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff800ca81e&gt;] __rmqueue+0x44/0xc6 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8000a939&gt;] get_page_from_freelist+0xbf/0x442 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff8000f46f&gt;] __alloc_pages+0x78/0x308 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff80025e20&gt;] find_or_create_page+0x32/0x72 Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff88aee445&gt;] filter_get_page+0x35/0x70 [obdfilter] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff88aeea81&gt;] filter_preprw_read+0x601/0xd30 [obdfilter] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff886ca230&gt;] LNetPut+0x730/0x840 [lnet] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887d5303&gt;] ptl_send_buf+0x3f3/0x5b0 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887dfc45&gt;] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff88af0f47&gt;] filter_preprw+0x1d97/0x1e00 [obdfilter] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887d5aa8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887ddc3a&gt;] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Feb 29 16:41:17 oss03 kernel: [&lt;ffffffff887d5aa8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff88a96ac3&gt;] ost_brw_read+0xb33/0x1a70 [ost] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887d9ed5&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887da088&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff88a9f363&gt;] ost_handle+0x2e73/0x55b0 [ost] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8872bd30&gt;] class_handle2object+0xe0/0x170 [obdclass] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8879419a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff88799168&gt;] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887e96b9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887e9e15&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887eada6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff887e9e40&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Feb 29 16:41:18 oss03 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 </pre> </div></div> <p>log files on OSS afer remove "client == LDLM_NAMESPACE_CLIENT"</p> <p>The same problem happened again.</p> <p>In this site, the client memory size is 96GB, each OSS's memory 24GB and they have 4 OSSs.<br/> We have enabled lru_resize on this client and it seems that OSS is getting memory pressure since OSS's memory is not reclaimed quickly.</p> <p>We just disabled lru_resize and used fixed lru_size and we haven't been seen same issue so far.</p> <p>We are going to be more testing on this configuration if the problem is gone completely.</p> <p>Hm, I guess there's some balancing to do on the server side to better reclaim locks preemptively before OOM situation arises.</p> <p>Hello, Oleg, </p> <p>The problem seems not to be fixed yet.. we are still getting some many call traces on OSSs even disable dynamic lru_size..<br/> I just uploaded the recent OSS's log files on ftp site. uploads/<a href="https://jira.whamcloud.com/browse/LU-1107" title="OSS dumps Call traces occasionally" class="issue-link" data-issue-key="LU-1107"><del>LU-1107</del></a>/2012-06-20.tar.gz</p> <p>Please have a look at them and investigate.</p> <p>Thanks! </p> <p>We hit this bug again today, but it caused 80+ clients to be unable to reconnect for a while. The customer restarted the machine, so it didn't finish trying to do the shrink. There was, however, an earlier event on the 23rd of August and that took over 3 hours <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/warning.png" height="16" width="16" align="absmiddle" alt="" border="0"/> to resolve. This is Lustre 1.8.6 (gce5e033)</p> <p>I'll try to get the lustre-logs that were dumped. In the meantime, I'll upload the oss log. It's worth looking at just to see this:<br/> Aug 23 15:53:37 oss2 kernel: Lustre: Service thread pid 7842 completed after 11067.64s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).</p> <p><img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Kernel log from OSS2, the one that hung</p> <p>This looks very similar to <a href="https://jira.whamcloud.com/browse/LU-1128" title="Complete investigation of the LDLM pool shrinker and SLV handling" class="issue-link" data-issue-key="LU-1128"><del>LU-1128</del></a>, and therefore <a href="https://jira.whamcloud.com/browse/LU-1535" title="LustreError: 1843:0:(mds_open.c:1645:mds_close()) " class="issue-link" data-issue-key="LU-1535"><del>LU-1535</del></a>. Can someone take a look at whether or not it's the same issue?</p> <p>Lai could you please comment?</p> <p>Hmm, IMO it's the same issue as <a href="https://jira.whamcloud.com/browse/LU-1128" title="Complete investigation of the LDLM pool shrinker and SLV handling" class="issue-link" data-issue-key="LU-1128"><del>LU-1128</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1535" title="LustreError: 1843:0:(mds_open.c:1645:mds_close()) " class="issue-link" data-issue-key="LU-1535"><del>LU-1535</del></a>, Kit, could you verify that patch is not included in your code?</p> <p>Hi Lai, we are not carrying the patch in <a href="https://jira.whamcloud.com/browse/LU-1535" title="LustreError: 1843:0:(mds_open.c:1645:mds_close()) " class="issue-link" data-issue-key="LU-1535"><del>LU-1535</del></a>, it looks like it's targeted for a 1.8.9 release. I'll get a build together for the customer to test on. </p> <p>Close old ticket.</p> Development End date Fri, 25 Jan 2013 13:00:19 +0000 Rank 1|hzw1gf: Rank (Obsolete) 10317 Severity Start date Wed, 15 Feb 2012 13:00:19 +0000