https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1444 Lustre <p>We have a large cluster installation where the /home directories are Lustre mounts. Occasionally (roughly once a week or so), we see the OSS lock up - this usually manifests itself as a hang doing 'df -h' or the like.</p> <p>Before the crash, this is what we see on the OSS:</p> <p> May 24 21:29:11 oss0 kernel: LustreError: 5460:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (<del>16) req@ffff810311398c00 x1396618176669269/t0 o8</del>&gt;2bf3e0b9-b782-cd24-0e06-8141a03901ff@NET_0x500000a370002_UUID:0/0 lens 368/264 e 0 to 0 dl 1337909450 ref 1 fl Interpret:/0/0 rc -16/0<br/> May 24 21:29:11 oss0 kernel: Lustre: 5475:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1396618176668701 took longer than estimated (756+139s); client may timeout. req@ffff810322b3a000 x1396618176668701/t0 o101-&gt;2bf3e0b9-b782-cd24-0e06-8141a03901ff@NET_0x500000a370002_UUID:0/0 lens 296/352 e 1 to 0 dl 1337909211 ref 1 fl Complete:/0/0 rc 0/0<br/> May 24 21:29:11 oss0 kernel: Lustre: lustre-OST0004: slow parent lock 892s due to heavy IO load<br/> May 24 21:29:11 oss0 kernel: Lustre: Skipped 2 previous similar messages<br/> May 24 21:29:11 oss0 kernel: Lustre: Service thread pid 5475 completed after 895.08s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> May 24 21:29:11 oss0 kernel: Lustre: lustre-OST0004: slow preprw_write setup 892s due to heavy IO load</p> <p>The -16 error seems to show up every time this happens. The OSS itself is using an Adaptec 6805. The arrays all show as optimal.</p> <p>The only fix we've found thus far is to reboot the OSS, then mount the OSS volumes (we don't have it set to automount on boot). Then, wait about 10 minutes for Lustre to recover. In the past, we've been able to pinpoint the problematic OSS (we have 2) by looking for unusual load via 'uptime', and then confirming -16 in /var/log/messages. </p> <p>Any assistance would be extremely helpful. Please let me know if you need any further information. Thanks. </p> CentOS 5.8; kernel 2.6.18-238.12.1.el5_lustre.g266a955 LU-1444

@@@ processing error (-16)

Bug Critical Resolved Cannot Reproduce Niu Yawei adam contois Tue, 29 May 2012 19:02:59 +0000 Tue, 19 Jun 2012 02:03:42 +0000 Tue, 19 Jun 2012 02:03:42 +0000 Lustre 1.8.6 0 2 <p>Niu</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, adam</p> <blockquote> <p>Before the crash, this is what we see on the OSS</p></blockquote> <p>Is the OSS locked up or crashed? Is there any other message dumped on crash?</p> <p>There are quite a few tickets for slow IO issue: <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a>, <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>, <a href="https://jira.whamcloud.com/browse/LU-410" title="Performance concern with Shrink file_max_cache_size to alleviate the memory pressure of OST patch for LU-15 " class="issue-link" data-issue-key="LU-410"><del>LU-410</del></a>. Disabling read only &amp; write through cache on OSS could alleviate the situation, is the read only &amp; write through cache on OSS is disabled? (you can veifty/change them in /proc/fs/$FSNAME/obdfilter/$OSTNAME/read_cache_enable and /proc/fs/$FSNAME/obdfilter/$OSTNAME/writethrough_cache_enable)</p> <p>Thanks.</p> <p>Hello Niu,</p> <p>I'm checking on the what exactly happens on the OSS, so I'll get back to you on that. </p> <p>read_cache_enable and writethrough_cache_enable both return true.</p> <p>Hi again Niu,</p> <p>When this has occurred, only Lustre becomes unavailable (filesystem commands hang) and the load on the OSS systems goes up.</p> <p>We did have one occurrence of the OSS being unreachable, but I'm not sure if it crashed or was under high load and unresponsive - we had to reboot it quickly to get users up and running again.</p> <p>Thanks.</p> <p>Thanks, Adam. Could you try to disable the read only cache &amp; write through cache on OSS to see if it can still be reproduced? Is the quota enabled? (I want to know if it's <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a>)</p> <p>Hi Niu,</p> <p>We would be happy to disable read and write cache - however, this is a production system with important data etc. So, we need specific instructions. Do we just echo 0 to those /proc files on the OSS'es only? Will this take effect immediately, or does it require a reboot? Furthermore, what are the potential negative ramifications of doing this? And (last question), can we try this on a single OSS, or would they all need to be changed? Thanks! </p> <p>You can either disable it by config log or write proc file directly, both take effect immediately, no reboot required.</p> <p>Run following command on MGS to disable them for certain OST permanently:</p> <p>lctl conf_param $OSTNAME.ost.read_cache_enable=0<br/> lctl conf_param $OSTNAME.ost.writethrough_cache_enable=0</p> <p>You can try this only for the problematical OSS, and there should be no visible impact on the customer application.</p> <p>Hi Niu,</p> <p>We have disabled the read and writethrough cache. So far, we haven't had further issues. You can close out the ticket, and I'll go ahead and reopen it if necessary. Thanks for your help.</p> <p>Thanks, adam.</p> Development Rank 1|hzvgx3: Rank (Obsolete) 6390 Severity