https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4875 Lustre <p>We have 2 OSS servers in HA with corosync. Each OSS has 12 OSTs mounted in failover. While working intermittently OSS server get reboots frequently, This is affecting the availability of file system badly.</p> LU-4875

We have 2 OSS server in HA and two MDS in HA , On each OSS 12 OSTs are mounted per OSS with faiover. OSS servers get reboots while working

Bug Critical Resolved Incomplete John Fuchs-Chesney Pankaj Sharma Wed, 9 Apr 2014 17:05:30 +0000 Wed, 29 Nov 2017 21:15:15 +0000 Tue, 19 May 2015 00:33:10 +0000 Lustre 2.2.0 0 7 <p>SOS report from system which contains all configuration and log files can be downloaded from <a href="https://ftp.usa.hp.com/hprc" class="external-link" target="_blank" rel="nofollow noopener">https://ftp.usa.hp.com/hprc</a> or ftp.usa.hp.com . Username is lustre and password is 7ui$HY6x.</p> <p>var/log/messages from both OSS are attached</p> <p>dmesg from both OSS servers are attached</p> <p>ost.threads_max counts are set to 256 and ost_io.threads_max are set to 256 as well .monitor timeout is 60 sec and dc-deadtime is 30 sec . Thanks</p> <p>I don't know what .xz files are, so I cannot look at them. The dmesg and messages files do not list how much RAM is on these nodes, nor what type of RAID you are using. Is it MD software RAID?</p> <p>My first guess would be that with 12 very large OSTs (I see 180 disks) on the node that it is just running out of memory. </p> <p>Thanks Andreas for prompt reply.<br/> Each OSS server has 32 GB RAM.<br/> We are using hardware RAID 5. Each OST consists of 11 x 300 GB SAS Disks in RAID 5. We have such 12 OSTs on each OSS. <br/> Yes probably you may be right that they may be running out of memory but how can we make sure of that, is there anything in logs or we can monitor in Lustre thru some debug option that it is running out of memory. If you need any other log then do let me know.<br/> I will extract .xz files and upload HA configuration files as .tar/.zip</p> <p>Do we have any parameters in Lustre thru which we can restrict running out of memory. As we have already reduced ost_io.threads_max to 256</p> <p>HA failover configuration file</p> <p>sar file from OSS1 for last 3 days are uploaded which can give us some idea for cpu utilization, I/O wait etc.</p> <p>we have noticed following in var/log/messages " max_child_count reached, postponing execution of operation monitor on ocf::Filesystem "</p> <p>Do this has some relation with reboot, if yes then what exactly this means</p> <p>HA configuration file - corosync.conf.txt and oos1-cibxml.txt are uploaded</p> <p>You should collect the messages from the console, which is best done by connecting via serial port to the servers. That will hopefully tell you exactly what is going wrong at the time of failure. </p> <p>What is the size of the journal on each OST? </p> <p>uploaded the last 2 days sar file from OSS2 server</p> <p>Please find OST detail below</p> <p><span class="error">&#91;root@homeoss1 ~&#93;</span># tune2fs -l /dev/mapper/mpathg<br/> tune2fs 1.42.7.wc2 (07-Nov-2013)<br/> device /dev/dm-6 mounted by lustre per /proc/fs/lustre/obdfilter/home-OST0006/mntdev<br/> Filesystem volume name: home-OST0006<br/> Last mounted on: /<br/> Filesystem UUID: 5a3ea3b2-568e-4062-a13b-ec5f121c0bd1<br/> Filesystem magic number: 0xEF53<br/> Filesystem revision #: 1 (dynamic)<br/> Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent mmp flex_bg sparse_super large_file huge_file uninit_bg dir_nlink<br/> Filesystem flags: signed_directory_hash<br/> Default mount options: user_xattr acl<br/> Filesystem state: clean<br/> Errors behavior: Continue<br/> Filesystem OS type: Linux<br/> Inode count: 11435008<br/> Block count: 731811520<br/> Reserved block count: 36590576<br/> Free blocks: 583168103<br/> Free inodes: 11013734<br/> First block: 0<br/> Block size: 4096<br/> Fragment size: 4096<br/> Reserved GDT blocks: 848<br/> Blocks per group: 32768<br/> Fragments per group: 32768<br/> Inodes per group: 512<br/> Inode blocks per group: 32<br/> RAID stripe width: 256<br/> Flex block group size: 256<br/> Filesystem created: Fri Mar 28 01:16:44 2014<br/> Last mount time: Wed Apr 9 16:22:44 2014<br/> Last write time: Wed Apr 9 16:22:44 2014<br/> Mount count: 51<br/> Maximum mount count: -1<br/> Last checked: Fri Mar 28 01:16:44 2014<br/> Check interval: 0 (&lt;none&gt;)<br/> Lifetime writes: 593 GB<br/> Reserved blocks uid: 0 (user root)<br/> Reserved blocks gid: 0 (group root)<br/> First inode: 11<br/> Inode size: 256<br/> Required extra isize: 28<br/> Desired extra isize: 28<br/> Journal inode: 8<br/> Default directory hash: half_md4<br/> Directory Hash Seed: 5cd731f7-67c3-4db6-9e7c-21db7e829749<br/> Journal backup: inode blocks<br/> MMP block number: 9734<br/> MMP update interval: 5</p> <p>we have not given any specific journal size while formatting, its default, have shared above OST info from which if you can make out.</p> <p>Hi Andreas, any update further on the logs and inputs we provided. Pls help we are in difficult position here.</p> <p>Regards,</p> <p>Pankaj</p> <p>The above information does not include the journal size, but if you used the default then it is 400MB and 12 journals are 4800MB and not enough use up the 32GB of RAM as I thought. </p> <p>You really need to post the serial console logs from the time of a crash. /var/log/messages and dmesg from a running system do not contain the information from the time of failure. </p> <p>Hi Andreas,</p> <p>At reboot following error appears at console <br/> "Message from syslogd@homeoss1 at Apr 15 12:46:34 ...<br/> kernel:LustreError: 7571:0:(ost_handler.c:1689:ost_prolong_lock_one()) ASSERTION( lock-&gt;l_req_mode == lock-&gt;l_granted_mode ) failed:</p> <p>Message from syslogd@homeoss1 at Apr 15 12:46:34 ...<br/> kernel:LustreError: 7571:0:(ost_handler.c:1689:ost_prolong_lock_one()) LBUG</p> <p>Message from syslogd@homeoss1 at Apr 15 12:46:34 ...<br/> kernel:Kernel panic - not syncing: LBUG<br/> "</p> <p>Also just to reduce the load we were trying to reduce the thread count on OSS, but while setting thread count to 128 on OSS2 we are getting following error<br/> <span class="error">&#91;root@homeoss2 ~&#93;</span># lctl set_param ost.OSS.ost.threads_max=128<br/> ost.OSS.ost.threads_max=128<br/> error: set_param: writing to file /proc/fs/lustre/ost/OSS/ost/threads_max: Numerical result out of range </p> <p>Also during last reboot on 12 April 14, we have observed following error messages, I f we can relate something with this</p> <p> Apr 12 02:03:55 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1397248434/real 1397248435&#93;</span> req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248441 ref 2 fl Rpc:X/0/ffffffff rc 0/-1<br/> Apr 12 02:03:55 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1397248435/real </p> <p>Apr 12 02:03:57 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1397248437/real 1397248437&#93;</span> req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248444 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:03:57 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 73468 previous similar messages<br/> Apr 12 02:03:59 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1397248439/real 1397248439&#93;</span> req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248446 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:03:59 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 162386 previous similar messages<br/> Apr 12 02:04:03 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1397248443/real 1397248443&#93;</span> req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248450 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:04:03 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 296701 previous similar messages</p> <p>1397248467] req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248474 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:04:27 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 1301167 previous similar messages<br/> Apr 12 02:04:59 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1397248499/real 1397248499&#93;</span> req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248506 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:04:59 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 2602376 previous similar messages<br/> Apr 12 02:06:03 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1397248563/real </p> <p>1397248563] req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248570 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:06:03 homeoss1 kernel: Lustre: 7390:0:(client.c:1788:ptlrpc_expire_one_request()) Skipped 5199842 previous similar messages<br/> Apr 12 02:07:14 homeoss1 kernel: Lustre: Service thread pid 7390 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:</p> <p>Apr 12 02:07:14 homeoss1 kernel: <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Apr 12 02:07:14 homeoss1 kernel:<br/> Apr 12 02:07:14 homeoss1 kernel: LustreError: dumping log to /tmp/lustre-log.1397248634.7390<br/> Apr 12 02:07:25 homeoss1 kernel: Lustre: home-OST0003: haven't heard from client c0bee620-b606-8adf-dadd-0d895330a3fd (at 10.2.1.252@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88090ff17000, cur 1397248645 expire 1397248495 last 1397248418<br/> Apr 12 02:07:25 homeoss1 kernel: LustreError: 7390:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8804e2e86c00 x1464903367127640/t0(0) o106-&gt;home-OST0005@10.2.1.252@o2ib:15/16 lens 296/232 e 0 to 1 dl 1397248652 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 12 02:07:25 homeoss1 kernel: LustreError: 138-a: home-OST0005: A client on nid 10.2.1.252@o2ib was evicted due to a lock glimpse callback time out: rc -4<br/> Apr 12 02:07:25 homeoss1 kernel: Lustre: Service thread pid 7390 completed after 210.90s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).</p> <p>Apr 12 02:28:07 homeoss1 crmd: <span class="error">&#91;5195&#93;</span>: notice: run_graph: Transition 256 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-253.bz2): Complete<br/> Apr 12 02:28:07 homeoss1 crmd: <span class="error">&#91;5195&#93;</span>: info: te_graph_trigger: Transition 256 is now complete<br/> Apr 12 02:28:07 homeoss1 crmd: <span class="error">&#91;5195&#93;</span>: info: notify_crmd: Transition 256 status: done - &lt;null&gt;<br/> Apr 12 02:28:07 homeoss1 crmd: <span class="error">&#91;5195&#93;</span>: info: do_state_transition: State transition S_TRANSITION_ENGINE -&gt; S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] <br/> Apr 12 02:28:07 homeoss1 crmd: <span class="error">&#91;5195&#93;</span>: info: do_state_transition: Starting PEngine Recheck Timer<br/> Apr 12 02:31:51 homeoss1 cib: <span class="error">&#91;5191&#93;</span>: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min<br/> Apr 12 02:38:25 homeoss1 kernel: imklog 4.6.2, log source = /proc/kmsg started.<br/> Apr 12 02:38:25 homeoss1 rsyslogd: <span class="error">&#91;origin software=&quot;rsyslogd&quot; swVersion=&quot;4.6.2&quot; x-pid=&quot;7609&quot; x-info=&quot;http://www.rsyslog.com&quot;&#93;</span> (re)start</p> <p>Uploaded log messages from OSS1 shows overloaded and hung threads as well. Can this be cause of reboot.</p> <p>As per the logs of 12 April , system rebooted at 2:28:07 prior to this at 02:07 it shows overload but what exactly this mean we don't know as CPU utilization was not high.</p> <p>Pls have a look at above facts and help us </p> <p>Hi Andreas,</p> <p>Please update.</p> <p>Regards,</p> <p>Pankaj</p> <p>Just wanted to clarify that this is NOT DDN+Intel supported Lustre and neither it is on DDN hardware. IIT Kanpur is DDN+Intel supported customer but for another system they procured. Not this one. </p> <p>I will watch this.<br/> ~ jfc.</p> <p>I'm marking this as resolved/incomplete because we have not been able to get sufficient appropriate information to debug the problem any further and the ticket is now more than six months old.</p> <p>~ jfc.</p> Development Rank 1|hzwjpb: Rank (Obsolete) 13484 Severity