https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1503 Lustre <p>Dear Support,<br/> we experienced some problem with our Lustre FS. <br/> Our users complained that at the following times their jobs were killed due IO errors.</p> <p>Saturday 09 June 05:58 <br/> Monday 11 June 12:47</p> <p>We collected the logs from the servers and clients side and actually we saw a lot of messages/errors that we have problem to "decode".<br/> Could you please help us to undestand why this problem arise ?<br/> In the specific we don't understand if it's really a overload problem related to the hardware or configuration we used otherwise some congestion/bug issue...</p> <p>Usually the FS seems to work correctly but suddently the log fill up of these messages.</p> <p>Regards<br/> Nicola</p> ----------------------------------------------------------------------------------------------------<br/> ## MDS HW ##<br/> ---------------------------------------------------------------------------------------------------- <br/> Linux XXXX.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 16 <br/> Vendor ID: AuthenticAMD <br/> CPU family: 16 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> ---<br/> MDT LSI 5480 Pikes Peak <br/> SSDs SLC <br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> <br/> ----------------------------------------------------------------------------------------------------<br/> ## OSS HW ##<br/> ---------------------------------------------------------------------------------------------------- <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 32 <br/> Vendor ID: GenuineIntel <br/> CPU family: 6 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> ---<br/> OSTs ---&gt; LSI 7900 SATA Disks<br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> <br/> ----------------------------------------------------------------------------------------------------<br/> ## Router nodes ##<br/> ---------------------------------------------------------------------------------------------------- <br/> 12 Cray XE6 Service nodes as router nodes - IB 40Gb/s <br/> ----------------------------------------------------------------------------------------------------<br/> <br/> <br/> ----------------------------------------------------------------------------------------------------<br/> ## Clients ##<br/> ---------------------------------------------------------------------------------------------------- <br/> ~ 1500 Cray XE6 nodes - Lustre 1.8.6 <br/> ----------------------------------------------------------------------------------------------------<br/> <br/> <br/> ----------------------------------------------------------------------------------------------------<br/> ## LUSTRE Config ##<br/> ---------------------------------------------------------------------------------------------------- <br/> 1 MDS + 1 fail over (MDT on SSD array)<br/> 12 OSSs - 6 OSTs per OSS (72 OSTs)<br/> <br/> Luster Servers ---&gt; 2.2.51.0<br/> Lustre Clients ---&gt; 1.8.6 (~1500 nodes) / 2.2.51.0 (~20 nodes)<br/> ---------------------------------------------------------------------------------------------------- LU-1503

Clients application IO errors and overloaded system messages

Bug Major Resolved Fixed Cliff White Nicola Bianchi Mon, 11 Jun 2012 10:39:21 +0000 Mon, 20 Jan 2014 18:09:39 +0000 Mon, 20 Jan 2014 18:09:39 +0000 Lustre 2.2.0 0 9 <p>Do you know what the load factor was on the servers when the outages occurred? It appears you are having client evictions, due to server timeouts.</p> <p>Cliff,<br/> it seems that there was more load than usual on the Lustre servers ... <br/> Could be interesting understand why because the HW behind is pretty powerful (12 brand new Sandy Bridge OSSs and 6 LSI 7900 Controllers with 8Gbit FC interfaces).</p> <p>Could be some configuration problem?</p> <p>Regards<br/> Nicola</p> <p>/modprobe.conf.local of router nodes on our Cray XE system:</p> <p>---------------------------------------------------------<br/> options dvsipc_lnet lnd_name=gni1<br/> options qla2xxx ql2xfailover=0<br/> options libcfs libcfs_panic_on_lbug=1<br/> options lnet ip2nets="gni 172.27.<b>.</b>; o2ib2 148.187.7.*"<br/> options lnet routes="gni 148.187.6.<span class="error">&#91;71-82&#93;</span>@o2ib2; o2ib2 <span class="error">&#91;220,226,270,304,394,436,474,484,1364,1386,1476,1530&#93;</span>@gni"<br/> options lnet check_routers_before_use=1 router_ping_timeout=5<br/> options lnet dead_router_check_interval=60 live_router_check_interval=60<br/> options kptllnd max_nodes=5000 credits=2048 timeout=250</p> <ol> <li> <ol> <li>Enable MSI for Mellanox ConnectX HCAs<br/> options mlx4_core msi_x=1<br/> ---------------------------------------------------------</li> </ol> </li> </ol> <p>Modprobe.conf.local of a client that mount the file system on our Cray XE:<br/> ---------------------------------------------------------<br/> options lnet ip2nets="gni1,gni 172.27.<b>.</b> ;o2ib 148.187.6.<b>;o2ib2 148.187.7.</b>"<br/> options lnet routes="gni1 148.187.6.<span class="error">&#91;142,140,141,146&#93;</span>@o2ib; o2ib <span class="error">&#91;62,174,967,1110&#93;</span>@gni1; gni 148.187.7.<span class="error">&#91;71-82&#93;</span>@o2ib2; o2ib2 <span class="error">&#91;220,226,270,304,394,436,474,484,1364,1386,1476,1530&#93;</span>@gni"<br/> options lnet check_routers_before_use=1 router_ping_timeout=5<br/> options lnet dead_router_check_interval=60 live_router_check_interval=60<br/> options qla2xxx ql2xlogintimeout=0<br/> options ost oss_num_threads=256<br/> options libcfs libcfs_panic_on_lbug=1<br/> options dvsipc_lnet lnd_name=gni1<br/> ---------------------------------------------------------</p> <p>modprobe.conf of a server:<br/> ---------------------------------------------------------<br/> options lnet networks="o2ib2(ib0)"<br/> options lnet routes="gni 148.187.7.<span class="error">&#91;71-82&#93;</span>@o2ib2"<br/> options lnet check_routers_before_use=1<br/> options lnet router_ping_timeout=5<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> ---------------------------------------------------------</p> <p>I don't think so, with configuration issues things tend to Not Work. I think this may be network-load related, are you sure the network is healthy? What does load look like on the router nodes?</p> <p>Cliff,<br/> thanks for the quick answer.<br/> For what we saw the 12 routers nodes seems to be pretty quiet but we will monitor the situation.</p> <p>For the infiniband network part we are checking if all is working well, so far the common IB tests reports a good shape but we see some strange behavior for clients that directly mount the FS through IB. In the logs we found messages like this:</p> <p> <span class="error">&#91;163598.503276&#93;</span> LustreError: 11-0: an error occurred while communicating with 148.187.7.106@o2ib2. The ost_connect operation failed with -16<br/> <span class="error">&#91;163598.503287&#93;</span> LustreError: Skipped 1 previous similar message</p> <p>...and we figured out that some common operation like untar a 2GB file sometime hang in conjunction the messages above. </p> <p>Any advice is welcome.</p> <p>Regards<br/> Nicola</p> <p>The -EBUSY would indicate that the OST was in recovery, or disconnecting/reconnecting. You should not see that message in normal operation. Are there any 'slow IO' messages in your logs? <br/> You need to check the logs for 148.187.7.106 and see what was happening on that server when the client reported the -EBUSY (-16)</p> <p>I would suggest running lnet_selftest to verify your network help, there are example scripts in the Lustre Manual. </p> <p>Cliff,<br/> in the logs for 148.187.7.106 (weisshorn06) we got this kind of messages:</p> <p>-----------------------------------------------------------------------------------<br/> Jun 14 09:50:47 weisshorn06 kernel: Lustre: scratch-OST0041: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting<br/> Jun 14 09:50:47 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages<br/> Jun 14 09:50:47 weisshorn06 kernel: Lustre: scratch-OST0041: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 11 active R<br/> PCs<br/> Jun 14 09:50:47 weisshorn06 kernel: Lustre: Skipped 1 previous similar message<br/> Jun 14 09:50:47 weisshorn06 kernel: LustreError: 6370:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc <del>107 req@ffff88101b3b9050 x1403838171590959/t0(0) o4</del>&gt;1d3575c8-<br/> d2c3-3917-0d4b-ae0b2c9af8d8@148.187.6.237@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1339660290 ref 1 fl Interpret:/0/0 rc 0/0<br/> Jun 14 09:50:47 weisshorn06 kernel: Lustre: scratch-OST0041: Bulk IO write error with 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2), client will retry: rc -107<br/> Jun 14 09:50:47 weisshorn06 kernel: LustreError: 6370:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 3 previous similar messages<br/> Jun 14 09:50:47 weisshorn06 kernel: LustreError: 8029:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc <del>107 req@ffff880c9942fc00 x1403838171590983/t0(0) o4</del>&gt;1d3575c8-<br/> d2c3-3917-0d4b-ae0b2c9af8d8@148.187.6.237@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1339660290 ref 1 fl Interpret:/0/0 rc 0/0<br/> Jun 14 09:50:47 weisshorn06 kernel: LustreError: 8029:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 4 previous similar messages<br/> Jun 14 09:51:11 weisshorn06 kernel: Lustre: scratch-OST0041: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting<br/> -----------------------------------------------------------------------------------</p> <p>I attach also the complete log that contain all the messages from all the OSS/MDS.</p> <p>Regards<br/> Nicola</p> <p>Cliff,<br/> about the lnet_selftest we run a batch script like in the manual and we got this:</p> <p>-------------------------------------------------------------------------------------<br/> SESSION: read/write TIMEOUT: 300 FORCE: No<br/> 148.187.7.<span class="error">&#91;101-114&#93;</span>@o2ib2 are added to session<br/> 148.187.6.<span class="error">&#91;6,7,8&#93;</span>@o2ib2 are added to session<br/> 148.187.6.<span class="error">&#91;6,7,8&#93;</span>@o2ib2 are added to session<br/> Test was added successfully<br/> Test was added successfully<br/> bulk_rw is running now<br/> <span class="error">&#91;LNet Rates of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 1395 RPC/s Min: 1 RPC/s Max: 5802 RPC/s<br/> <span class="error">&#91;W&#93;</span> Avg: 1555 RPC/s Min: 1 RPC/s Max: 6653 RPC/s<br/> <span class="error">&#91;LNet Bandwidth of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 72.86 MB/s Min: 0.00 MB/s Max: 163.51 MB/s<br/> <span class="error">&#91;W&#93;</span> Avg: 163.19 MB/s Min: 0.00 MB/s Max: 855.02 MB/s<br/> <span class="error">&#91;LNet Rates of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 1450 RPC/s Min: 15 RPC/s Max: 5744 RPC/s<br/> <span class="error">&#91;W&#93;</span> Avg: 1610 RPC/s Min: 15 RPC/s Max: 6599 RPC/s<br/> <span class="error">&#91;LNet Bandwidth of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 105.09 MB/s Min: 0.00 MB/s Max: 240.24 MB/s<br/> <span class="error">&#91;W&#93;</span> Avg: 163.03 MB/s Min: 0.00 MB/s Max: 858.62 MB/s<br/> <span class="error">&#91;LNet Rates of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 1459 RPC/s Min: 0 RPC/s Max: 5800 RPC/s<br/> <span class="error">&#91;W&#93;</span> Avg: 1619 RPC/s Min: 0 RPC/s Max: 6657 RPC/s<br/> <span class="error">&#91;LNet Bandwidth of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 88.47 MB/s Min: 0.00 MB/s Max: 199.88 MB/s<br/> <span class="error">&#91;W&#93;</span> Avg: 163.60 MB/s Min: 0.00 MB/s Max: 858.06 MB/s<br/> <span class="error">&#91;LNet Rates of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 2946 RPC/s Min: 0 RPC/s Max: 7408 RPC/s<br/> <span class="error">&#91;W&#93;</span> Avg: 3104 RPC/s Min: 0 RPC/s Max: 8241 RPC/s<br/> <span class="error">&#91;LNet Bandwidth of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 85.92 MB/s Min: 0.00 MB/s Max: 192.92 MB/s<br/> <span class="error">&#91;W&#93;</span> Avg: 163.32 MB/s Min: 0.00 MB/s Max: 838.31 MB/s<br/> <span class="error">&#91;LNet Rates of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 1374 RPC/s Min: 0 RPC/s Max: 5569 RPC/s<br/> <span class="error">&#91;W&#93;</span> Avg: 1534 RPC/s Min: 0 RPC/s Max: 6419 RPC/s<br/> <span class="error">&#91;LNet Bandwidth of servers&#93;</span><br/> <span class="error">&#91;R&#93;</span> Avg: 70.01 MB/s Min: 0.00 MB/s Max: 162.32 MB/s<br/> <span class="error">&#91;W&#93;</span> Avg: 165.01 MB/s Min: 0.00 MB/s Max: 854.23 MB/s<br/> session is ended<br/> lustre_self_test.sh: line 17: 17805 Terminated lst stat servers<br/> -------------------------------------------------------------------------------------</p> <p>Still difficult for us to understand the exact meaning of the results but running multiple times we got always the same scores.<br/> At a glance: we run 3 clients (that time to time loses the connections) against the 14 nodes of the Lustre FS. </p> <p>Here the script we used:<br/> -------------------------------------------------------------------------------------<br/> #!/bin/bash<br/> export LST_SESSION=$$<br/> lst new_session read/write<br/> lst add_group servers 148.187.7.<span class="error">&#91;101-114&#93;</span>@o2ib2<br/> lst add_group readers 148.187.6.<span class="error">&#91;6,7,8&#93;</span>@o2ib2<br/> lst add_group writers 148.187.6.<span class="error">&#91;6,7,8&#93;</span>@o2ib2<br/> lst add_batch bulk_rw<br/> lst add_test --batch bulk_rw --from readers --to servers \<br/> brw read check=simple size=1M<br/> lst add_test --batch bulk_rw --from writers --to servers \<br/> brw write check=full size=4K<br/> ##start running<br/> lst run bulk_rw<br/> ##display server stats for 30 seconds<br/> lst stat servers &amp; sleep 30; kill $!<br/> ##tear down<br/> lst end_session<br/> -------------------------------------------------------------------------------------</p> <p>Regards<br/> Nicola</p> <p>Those result tell you how much IO you can generate from three clients. How many clients do you normally run?<br/> I would suggest doing a test with <em>all</em> your clients, and monitor your network hardware for errors. <br/> Also, please list the distribution and Lustre version for your clients and servers. </p> <p>From the router logs, it appears the gni side is not especially happy, there are timeouts, mis-routes:</p> <p>Jun 11 12:29:48 nid00484 kernel: LNet: 12032:0:(gnilnd_conn.c:1872:kgnilnd_reaper_dgram_check()) GNILND_DGRAM_REQ datagram to 385@gni timed out @ 63s dgram 0xffff8803cf3e26c8 state GNILND_DGRAM_POSTED conn 0xffff880406d64400<br/> Jun 11 12:29:48 nid00484 kernel: HWERR<span class="error">&#91;2899&#93;</span>:0x0b11:SSID Detected Misrouted Packet:Info1=0x8001025000014281:Info2=0x0:Info3=0x2072<br/> Jun 11 12:29:51 nid00484 kernel: LNet: could not send to 385@gni due to connection setup failure after 66 seconds<br/> Jun 11 12:29:51 nid00484 kernel: LNet: Skipped 12 previous similar messages<br/> Jun 11 12:29:51 nid00484 kernel: LNet: 12032:0:(gnilnd_conn.c:790:kgnilnd_process_dgram()) hardware timeout for connect to 385@gni after 0 seconds. Is node dead?<br/> Jun 11 12:29:51 nid00484 kernel: HWERR<span class="error">&#91;2900&#93;</span>:0x0b11:SSID Detected Misrouted Packet:Info1=0x8001025000014281:Info2=0x0:Info3=0x20cc<br/> Jun 11 12:30:12 nid00484 kernel: HWERR<span class="error">&#91;2901&#93;</span>:0x0b11:SSID Detected Misrouted Packet:Info1=0x8001025000014281:Info2=0x0:Info3=0x2126<br/> Jun 11 12:30:34 nid00484 kernel: HWERR<span class="error">&#91;2902&#93;</span>:0x0b11:SSID Detected Misrouted Packet:Info1=0x8001025000014281:Info2=0x0:Info3=0x2180<br/> Jun 11 12:30:56 nid00484 kernel: LNet: could not send to 385@gni due to connection setup failure after 65 seconds</p> <p>can you identify which node uses '385@gni' as it's address? Checking that node's logs might produce some useful information. </p> <p>Cliff,<br/> usually we run from about 1500 Cray XE6 nodes and ~20 standard Linux Infiniband nodes. </p> <p>Cheers<br/> Nicola</p> <p>Cliff,<br/> about the '385@gni':</p> <p> this node failed (HW Failure nid00385 c4-0c0s0n1) and for this reason you see the error.</p> <p>Cheers<br/> Nicola</p> <p>Cliff,<br/> we noticed that when on our 1500 nodes Cray machine we have congestion problems (due HW problem for instance) the Lustre servers suffer the situation, as you can see in the logs.</p> <p>There is some timeouts parameter that we can tune to mitigate the situation?</p> <p>We know that on the Cray side the time to reconfigure the routing due an error should not be so long but anyway it seems that something with the filesystem is still hanging ...</p> <p>Regards<br/> Nicola</p> <p>I will investigate, but most timeouts in lustre are auto-tuned. I am talking to our engineers. </p> <p>I didn't get chance to look into it, but seems it's gnilnd right? It's not in Lustre source tree yet, where can we find source code?</p> <p>Doug, I've added you to CC list, could you please look into it? I'm quite busy in recent a few days and have to attend a two days <br/> meeting since tomorrow.</p> <p>Hi Cliff,<br/> for your information, on weisshorn02 you can find in /var/log/cluster.log all the collected log from the entire cluster.<br/> And if you run the command 'ltop' you can have a look of the on line performance.</p> <p>What do you think about these parameters to pass to router/client/servers does them make any sense to you?</p> <ol> <li> <ol> <li>Router</li> </ol> </li> </ol> <p>options ko2iblnd timeout=100 peer_timeout=130<br/> options ko2iblnd credits=2048 ntx=2048<br/> options ko2iblnd peer_credits=126 concurrent_sends=63 peer_buffer_credits=128<br/> options kgnilnd credits=2048 peer_health=1</p> <p>options lnet check_routers_before_use=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet router_ping_timeout=50<br/> options lnet large_router_buffers=1024 small_router_buffers=16384</p> <ol> <li> <ol> <li>Server</li> </ol> </li> </ol> <p>options ko2iblnd timeout=100 peer_timeout=0 keepalive=30<br/> options ko2iblnd credits=2048 ntx=2048<br/> options ko2iblnd peer_credits=126 concurrent_sends=63</p> <p>options lnet avoid_asym_router_failure=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet check_routers_before_use=1</p> <ol> <li> <ol> <li>Client</li> </ol> </li> </ol> <p>options ko2iblnd timeout=100 peer_timeout=0 keepalive=30<br/> options ko2iblnd credits=2048 ntx=2048<br/> options ko2iblnd peer_credits=126 concurrent_sends=63</p> <p>options lnet avoid_asym_router_failure=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet check_routers_before_use=1</p> <p>The area of concern for me is this:<br/> #cat /proc/sys/lnet/peers</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nid refs state last max rtr min tx min queue .... 148.187.7.71@o2ib2 6 up 9999 8 8 8 5 -177 1824 148.187.7.72@o2ib2 7 up 9999 8 8 8 5 -177 1824 148.187.7.73@o2ib2 5 up 9999 8 8 8 6 -175 1216 148.187.7.74@o2ib2 6 up 9999 8 8 8 5 -177 1824 148.187.7.75@o2ib2 6 up 9999 8 8 8 5 -178 1824 148.187.7.76@o2ib2 8 up 9999 8 8 8 4 -177 2432 148.187.7.77@o2ib2 6 up 9999 8 8 8 5 -177 1824 148.187.7.78@o2ib2 7 up 9999 8 8 8 5 -178 1824 148.187.7.79@o2ib2 7 up 9999 8 8 8 4 -176 2432 148.187.7.80@o2ib2 7 up 9999 8 8 8 4 -176 2432 148.187.7.81@o2ib2 6 up 9999 8 8 8 5 -178 1824 148.187.7.82@o2ib2 7 up 9999 8 8 8 5 -178 1824 ... </pre> </div></div> <p>Those are the gni routers, and that is not especially normal. <br/> All of your options that have the word 'check' in them are good, normal. <br/> We think the peer_credits is high, we would advise </p> <p>options ko2iblnd peer_credits=16 concurrent_sends=16</p> <p>The 'min' parameter, when negative indicates the number of queued messages, the 'queue' indicates<br/> the number of bytes queued from that peer. So that is an indication things are backed up.</p> <p>I'm suspecting that the messages_router_node.log gives a good indication of what may be happening:</p> <p>At: <br/> 12:28:23 - problems communicating with one node is reported.<br/> 12:28:28 - hardware quiesce is reported. I'm not familiar with the gnilnd (or Gemini), but this implies to me that a hardware bit in Gemini has flipped telling software to back off...things are not good.<br/> 12:28:40 - All the gnilnd threads are now paused due to the hardware quiesce. At this point, timeouts and message drops are rampant. That would be due to everything being paused.<br/> 12:28:46 - All threads are back up and running again allowing traffic to flow.</p> <p>So, this is how I read what is going on (but a Gemini expert is needed to clarify):<br/> The node which has a hardware fault somehow triggers back pressure on the gnilnd of the router. This trigger the hardware to complain forcing the gnilnd to shut down all traffic on that interface for 6 seconds! In that time, all the IB queues which need to forward to that interface will back up possibly causing many critical message drops. It is hard to say how long after the 6 second lock down it takes for everything to recover.</p> <p>Dear Support,</p> <p>this morning: Jun 25 11:51:18 the MDS crashed due a kernel panic (weisshorn01) just after the crash for unknown reasons of the OSS weisshorn14.<br/> All logs are on weisshorn02:/var/log/cluster.log</p> <p>Regards<br/> Nicola </p> <p>Crash one:<br/> un 4 10:10:57 weisshorn01 kernel: LustreError: 4365:0:(mdd_object.c:635:mdd_big_lmm_get()) ASSERTION( ma-&gt;ma_lmm_size &gt; 0 ) failed:<br/> Jun 4 10:10:57 weisshorn01 kernel: LustreError: 4365:0:(mdd_object.c:635:mdd_big_lmm_get()) LBUG<br/> Jun 4 10:10:57 weisshorn01 kernel: Pid: 4365, comm: mdt_09<br/> Jun 4 10:10:57 weisshorn01 kernel:<br/> Jun 4 10:10:57 weisshorn01 kernel: Call Trace:<br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03c0915&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03c0e47&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bf29e3&gt;&#93;</span> mdd_big_lmm_get+0x433/0x4f0 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bfb9a0&gt;&#93;</span> ? mdd_get_md+0xa0/0x2d0 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bf34ee&gt;&#93;</span> __mdd_lmm_get+0x1ce/0x2c0 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bf6c09&gt;&#93;</span> mdd_attr_get_internal+0x249/0x770 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0d1279f&gt;&#93;</span> ? osd_object_read_lock+0x9f/0x140 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bf7188&gt;&#93;</span> mdd_attr_get_internal_locked+0x58/0x80 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0604af0&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x6d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c6b3c0&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x210 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0bf71ed&gt;&#93;</span> mdd_attr_get+0x3d/0xa0 <span class="error">&#91;mdd&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0d618bc&gt;&#93;</span> cml_attr_get+0x6c/0x160 <span class="error">&#91;cmm&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0625d84&gt;&#93;</span> ? lustre_msg_get_opc+0x64/0xa0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c77884&gt;&#93;</span> mdt_getattr_internal+0x294/0xd00 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c7adf5&gt;&#93;</span> mdt_getattr_name_lock+0xd25/0x1700 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa062607d&gt;&#93;</span> ? lustre_msg_buf+0x5d/0x60 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa064b756&gt;&#93;</span> ? __req_capsule_get+0x176/0x640 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0627d74&gt;&#93;</span> ? lustre_msg_get_flags+0x34/0x70 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c7bcdd&gt;&#93;</span> mdt_intent_getattr+0x2cd/0x4a0 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c79591&gt;&#93;</span> mdt_intent_policy+0x2d1/0x600 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa05eae69&gt;&#93;</span> ldlm_lock_enqueue+0x2f9/0x830 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa060c197&gt;&#93;</span> ldlm_handle_enqueue0+0x427/0xda0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c791d6&gt;&#93;</span> mdt_enqueue+0x46/0x130 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c7184d&gt;&#93;</span> mdt_handle_common+0x74d/0x1400 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0c725d5&gt;&#93;</span> mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa06333c1&gt;&#93;</span> ptlrpc_server_handle_request+0x3c1/0xcb0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03c14ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03cbef9&gt;&#93;</span> ? lc_watchdog_touch+0x79/0x110 <span class="error">&#91;libcfs&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa062d462&gt;&#93;</span> ? ptlrpc_wait_event+0xb2/0x2c0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff810519c3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa06343cf&gt;&#93;</span> ptlrpc_main+0x71f/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0633cb0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> Jun 4 10:10:57 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0633cb0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 4 10:12:28 weisshorn02 heartbeat: <span class="error">&#91;3260&#93;</span>: WARN: node weisshorn01.admin.cscs.ch: is dead</p> <p>Second crash:</p> <p>Jun 25 11:51:18 weisshorn01 kernel: LustreError: 3589:0:(ldlm_lock.c:831:ldlm_lock_decref_and_cancel()) ASSERTION( lock != ((void *)0) ) failed:<br/> Jun 25 11:51:18 weisshorn01 kernel: LustreError: 3589:0:(ldlm_lock.c:831:ldlm_lock_decref_and_cancel()) LBUG<br/> Jun 25 11:51:18 weisshorn01 kernel: Pid: 3589, comm: mgs_scratch_not<br/> Jun 25 11:51:18 weisshorn01 kernel:<br/> Jun 25 11:51:18 weisshorn01 kernel: Call Trace:<br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03bc915&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03bce47&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa05e92e1&gt;&#93;</span> ldlm_lock_decref_and_cancel+0x111/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b6448b&gt;&#93;</span> mgs_completion_ast_ir+0xfb/0x110 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0600810&gt;&#93;</span> ldlm_cli_enqueue_local+0x1f0/0x4d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b64390&gt;&#93;</span> ? mgs_completion_ast_ir+0x0/0x110 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa05ff940&gt;&#93;</span> ? ldlm_blocking_ast+0x0/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b6429f&gt;&#93;</span> mgs_revoke_lock+0x13f/0x230 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa05ff940&gt;&#93;</span> ? ldlm_blocking_ast+0x0/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b64390&gt;&#93;</span> ? mgs_completion_ast_ir+0x0/0x110 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa03c65c1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b7a4ec&gt;&#93;</span> mgs_ir_notify+0x11c/0x230 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff8105e7f0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff81007966&gt;&#93;</span> ? xen_timer_interrupt+0x16/0x1b0<br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b7a3d0&gt;&#93;</span> ? mgs_ir_notify+0x0/0x230 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b7a3d0&gt;&#93;</span> ? mgs_ir_notify+0x0/0x230 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffffa0b7a3d0&gt;&#93;</span> ? mgs_ir_notify+0x0/0x230 <span class="error">&#91;mgs&#93;</span><br/> Jun 25 11:51:18 weisshorn01 kernel: <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Jun 25 11:51:18 weisshorn01 kernel:<br/> Jun 25 11:51:18 weisshorn01 kernel: Kernel panic - not syncing: LBUG<br/> Jun 25 11:51:18 weisshorn01 kernel: Pid: 3589, comm: mgs_scratch_not Not tainted 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 #1</p> <p>The first LBUG appears to be this:<a href="http://jira.whamcloud.com/browse/LU-1384" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1384</a></p> <p>It seems that you are running a patched version of 2.2 because Lustre-2.2.51-2.6.32_220.7.1.el6_lustre.g9c8f747.x86_64_gd2c1a39.x86_64 indicates a version of Lustre taken from gerrit rather than an official release. Which issue did this address? Are you comfortable with how to apply patches to your Lustre version so that you can also add the fix for <a href="https://jira.whamcloud.com/browse/LU-1384" title="MDS Kernel Panic when trying to mount the lustre file system" class="issue-link" data-issue-key="LU-1384"><del>LU-1384</del></a>?</p> <p>Cliff,<br/> my colleague told me that we are using the version Lustre-2.2.51-2.6.32_220.7.1.el6_lustre.g9c8f747.x86_64_gd2c1a39.x86_64 because this was the only one that we were able to use on our sandy-bridge OSSes. With other versions we experienced instability or inability to boot the servers.</p> <p>About the patches I guess that with some procedure we should be able to get it done.<br/> Anyway, we have for sure to wait at least for our next maintenance window.</p> <p>Regards<br/> Nicola</p> <p>Cliff,<br/> regarding your suggestion for this parameter:</p> <p> options ko2iblnd peer_credits=16 concurrent_sends=16</p> <p>Currently we don't specified anything about that in our modprobe.conf, we use the configuration posted by Fabio Verzelloni on this date: 12/Jun/12 3:44 AM.<br/> The parameters you posted the "22/Jun/12 10:09 AM" , from a Fabio email, are suggestions found in a 3rd party manual... and we were wondering if some of these options could improve our situation.</p> <p>Regards <br/> Nicola</p> <p>We really would like you to try options ko2iblnd peer_credits=16 concurrent_sends=16<br/> We think that may improve the situation. <br/> The other options you have listed from the 3rd party manual will not help, and setting the large values for peer_credits and concurrent_sends in that list will likely make things worse. </p> <p>Please try our suggestion and report your results. </p> <p>Cliff,</p> <p>it's ok with you if we will put in the configuration the parameter you suggested in the next maintenance day we planned (4th July)?</p> <p>Regards<br/> Nicola</p> <p>Cliff,<br/> right now we are experiencing some performance issue on the filesystem.<br/> Users are complaining about a drop down of the performance and in fact I can see that a simple "ls /scratch/weisshorn" can take minutes to answer.</p> <p>In the logs there are a bunch of this messages:</p> <p>---------------------------------------------------------------------------------------------------<br/> Jun 26 13:50:59 weisshorn02 kernel: LustreError: 9825:0:(lov_obd.c:1068:lov_clear_orphans()) error in orphan recovery on OST idx 65/72: rc = -5<br/> Jun 26 13:50:59 weisshorn02 kernel: LustreError: 9825:0:(mds_lov.c:884:__mds_lov_synchronize()) scratch-OST0041_UUID failed at mds_lov_clear_orphans: -5<br/> Jun 26 13:50:59 weisshorn02 kernel: LustreError: 9825:0:(mds_lov.c:905:__mds_lov_synchronize()) scratch-OST0041_UUID sync failed -5, deactivating<br/> Jun 26 13:51:01 weisshorn02 /usr/sbin/cerebrod<span class="error">&#91;2756&#93;</span>: lmt_mysql: failed to connect to database<br/> Jun 26 13:51:04 weisshorn11 kernel: Lustre: 5917:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1340711408/real 1340711408&#93;</span> req@ffff880807301400 x1405364552083621/t0(0) o250-&gt;MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340711464 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Jun 26 13:51:04 weisshorn11 kernel: Lustre: 5917:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages<br/> Jun 26 13:51:07 weisshorn02 kernel: LustreError: 9835:0:(lov_obd.c:1068:lov_clear_orphans()) error in orphan recovery on OST idx 25/72: rc = -5<br/> Jun 26 13:51:07 weisshorn02 kernel: LustreError: 9835:0:(mds_lov.c:884:__mds_lov_synchronize()) scratch-OST0019_UUID failed at mds_lov_clear_orphans: -5<br/> Jun 26 13:51:07 weisshorn02 kernel: LustreError: 9835:0:(mds_lov.c:905:__mds_lov_synchronize()) scratch-OST0019_UUID sync failed -5, deactivating<br/> Jun 26 13:51:13 weisshorn04 kernel: Lustre: 5655:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1340711417/real 1340711417&#93;</span> req@ffff880feec76000 x1405364532655945/t0(0) o250-&gt;MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340711473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Jun 26 13:51:13 weisshorn04 kernel: Lustre: 5655:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages<br/> Jun 26 13:51:16 weisshorn02 /usr/sbin/cerebrod<span class="error">&#91;2756&#93;</span>: lmt_mysql: failed to connect to database<br/> ---------------------------------------------------------------------------------------------------</p> <p>We don't see any issue with the network, neither IB nor GNI.</p> <p>Machine load:<br/> ---------------------------------------------------------------------------------------------------<br/> weisshorn05: 14:01:32 up 5 days, 5:41, 0 users, load average: 3.12, 3.61, 3.85<br/> weisshorn14: 14:01:32 up 46 min, 0 users, load average: 0.02, 0.01, 0.00 &lt;----- out of production<br/> weisshorn03: 14:01:32 up 5 days, 5:41, 0 users, load average: 5.26, 4.41, 3.86<br/> weisshorn07: 14:01:32 up 5 days, 5:41, 0 users, load average: 3.21, 3.41, 3.95<br/> weisshorn04: 14:01:32 up 5 days, 5:41, 0 users, load average: 4.47, 5.05, 4.90<br/> weisshorn06: 14:01:32 up 5 days, 5:41, 0 users, load average: 4.32, 4.04, 4.08<br/> weisshorn08: 14:01:32 up 5 days, 5:40, 0 users, load average: 5.11, 4.73, 4.91<br/> weisshorn01: 14:01:32 up 1 day, 1:53, 0 users, load average: 0.10, 0.03, 0.01 &lt;---- out of production<br/> weisshorn09: 14:01:32 up 5 days, 5:41, 0 users, load average: 3.83, 4.29, 4.68<br/> weisshorn11: 14:01:32 up 5 days, 5:40, 0 users, load average: 3.29, 3.69, 3.73<br/> weisshorn12: 14:01:32 up 5 days, 5:40, 0 users, load average: 3.37, 3.26, 3.06<br/> weisshorn10: 14:01:32 up 5 days, 5:40, 0 users, load average: 4.26, 3.72, 3.36<br/> weisshorn13: 14:01:32 up 5 days, 5:40, 0 users, load average: 9.32, 8.07, 7.24<br/> weisshorn02: 14:01:32 up 5 days, 5:38, 5 users, load average: 0.26, 1.88, 2.58 &lt;--- MDS<br/> ---------------------------------------------------------------------------------------------------</p> <p>ltop screenshot:<br/> ---------------------------------------------------------------------------------------------------<br/> Filesystem: scratch<br/> Inodes: 185.812m total, 16.950m used ( 9%), 168.863m free<br/> Space: 503.720t total, 296.873t used ( 59%), 206.847t free<br/> Bytes/s: 0.323g read, 0.285g write, 2711 IOPS<br/> MDops/s: 245 open, 262 close, 14 getattr, 6 setattr<br/> 0 link, 1 unlink, 0 mkdir, 0 rmdir<br/> 0 statfs, 3 rename, 0 getxattr<br/> &gt;OST S OSS Exp CR rMB/s wMB/s IOPS LOCKS LGR LCR %cpu %mem %spc<br/> (12) F eisshorn13 1551 0 56 41 525 1156010 9 363 1 100 61<br/> (6) F eisshorn03 1551 0 28 28 127 560595 7 2 0 100 59<br/> (6) F eisshorn04 1551 0 37 36 110 614243 0 0 0 100 56<br/> (6) F eisshorn05 1551 0 37 25 182 621734 0 0 0 100 61<br/> (6) F eisshorn06 1551 0 22 21 374 597840 0 0 1 100 57<br/> (6) F eisshorn07 1551 0 22 16 234 653836 2 0 0 100 60<br/> (6) F eisshorn08 1551 0 24 31 103 713393 3 117 0 100 53<br/> (6) F eisshorn09 1551 0 26 25 344 580579 2 98 1 100 55<br/> (6) F eisshorn10 1551 0 28 27 445 595866 1 164 1 100 59<br/> (6) F eisshorn11 1551 0 24 19 169 565911 4 4 0 100 62<br/> (6) F eisshorn12 1551 0 27 24 98 696995 2 152 0 100 62<br/> ---------------------------------------------------------------------------------------------------</p> <p>cat /proc/sys/lnet/peers<br/> ---------------------------------------------------------------------------------------------------<br/> 148.187.7.71@o2ib2 3 up 9999 8 8 8 8 -611 0<br/> 148.187.7.72@o2ib2 3 up 9999 8 8 8 8 -592 0<br/> 148.187.7.73@o2ib2 3 up 9999 8 8 8 8 -633 0<br/> 148.187.7.74@o2ib2 3 up 9999 8 8 8 8 -608 0<br/> 148.187.7.75@o2ib2 3 up 9999 8 8 8 8 -617 0<br/> 148.187.7.76@o2ib2 3 up 9999 8 8 8 8 -642 0<br/> 148.187.7.77@o2ib2 3 up 9999 8 8 8 8 -618 0<br/> 148.187.7.78@o2ib2 3 up 9999 8 8 8 8 -626 0<br/> 148.187.7.79@o2ib2 3 up 9999 8 8 8 8 -636 0<br/> 148.187.7.80@o2ib2 3 up 9999 8 8 8 8 -630 0<br/> 148.187.7.81@o2ib2 3 up 9999 8 8 8 8 -636 0<br/> 148.187.7.82@o2ib2 3 up 9999 8 8 8 8 -635 0<br/> ---------------------------------------------------------------------------------------------------</p> <p>Regards<br/> Nicola</p> <p>You are reporting issues every day - i would NOT wait for the maint window to change the configuration. You could do a rolling failover and set this up without halting the filesystem, simply failover one node at a time.</p> <p>The current issue may be related to <a href="https://jira.whamcloud.com/browse/LU-1247" title="After recovery 7 of 16 OST were deactivated by MDS." class="issue-link" data-issue-key="LU-1247"><del>LU-1247</del></a>, am consulting engineering. I think we need to point you to a newer release, and should plan on a system upgrade on the 4th. </p> <p>Cliff,<br/> do I have to put this line only on the servers? (OSS and MDS)</p> <p> options ko2iblnd peer_credits=16 concurrent_sends=16</p> <p>Regards<br/> Nicola</p> <p>I am sorry, I was in error. The "options ko2iblnd peer_credits=16 concurrent_sends=16" needs to be set on all nodes, client and server. You would have to stop the clients for a moment to do this, the rolling failover idea could be used for servers, but likely better to do all at once.</p> <p>Cliff,<br/> is it a real problem if I begin to put that on the servers only?</p> <p> Nicola</p> <p>the second crash is <a href="https://jira.whamcloud.com/browse/LU-1259" title="Test failure on test suite conf-sanity, subtest test_41a" class="issue-link" data-issue-key="LU-1259"><del>LU-1259</del></a>, patch at <a href="http://review.whamcloud.com/2390" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2390</a></p> <p>This via email:<br/> Cliff,<br/> thanks a lot for the informations.</p> <p>Hence the next Wednesday Fabio, who read us in copy, will proceed with the downgrade to the 2.1.2 version. Since this afternoon I will be away for 7 weeks so he will take over this task ...</p> <p>For sure Fabio will come back to you about the details and the schedule for this intervention.</p> <p>------</p> <p>Okay, if you need help on the 4th, please email <br/> dutymanager@whamcloud.com in the event of problems during the upgrade.<br/> It is a US holiday, so some of us will not be available.</p> <p>And of course, the bits you need are available here: <a href="http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/" class="external-link" target="_blank" rel="nofollow noopener">http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/</a></p> <p>I just tried on a test machine to recreate the situation will happen on Wednesday, basically downgrade from 2.2 --&gt; 2.1.2 and I've got the following error message:</p> <p>console log</p> <p>Jul 02 16:08 <span class="error">&#91;root@wn47:~&#93;</span># mount -t lustre /dev/vg_root/mds /mnt/lustre<br/> mount.lustre: mount /dev/mapper/vg_root-mds at /mnt/lustre failed: Invalid argument<br/> This may have multiple causes.<br/> Are the mount options correct?<br/> Check the syslog for more info.</p> <p>messages</p> <p>Jul 2 16:10:17 wn47 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> Jul 2 16:10:17 wn47 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: <br/> Jul 2 16:10:17 wn47 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> Jul 2 16:10:17 wn47 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: <br/> Jul 2 16:10:17 wn47 kernel: Lustre: MGS MGS started<br/> Jul 2 16:10:17 wn47 kernel: Lustre: 8536:0:(ldlm_lib.c:933:target_handle_connect()) MGS: connection from bd7d7986-674e-84d2-d489-d6f9028b0d52@0@lo t0 exp (null) cur 1341238217 last 0<br/> Jul 2 16:10:17 wn47 kernel: Lustre: MGC10.10.65.47@tcp: Reactivating import<br/> Jul 2 16:10:17 wn47 kernel: Lustre: Enabling ACL<br/> Jul 2 16:10:17 wn47 kernel: Lustre: lustre-MDT0000: used disk, loading<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8540:0:(mdt_recovery.c:409:mdt_server_data_init()) lustre-MDT0000: unsupported incompat filesystem feature(s) 200<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8540:0:(obd_config.c:522:class_setup()) setup lustre-MDT0000 failed (-22)<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8540:0:(obd_config.c:1363:class_config_llog_handler()) Err -22 on cfg command:<br/> Jul 2 16:10:17 wn47 kernel: Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f <br/> Jul 2 16:10:17 wn47 kernel: LustreError: 15b-f: MGC10.10.65.47@tcp: The configuration from log 'lustre-MDT0000'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre.<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 15c-8: MGC10.10.65.47@tcp: The configuration from log 'lustre-MDT0000' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8458:0:(obd_mount.c:1192:server_start_targets()) failed to start server lustre-MDT0000: -22<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8458:0:(obd_mount.c:1738:server_fill_super()) Unable to start targets: -22<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8458:0:(obd_config.c:567:class_cleanup()) Device 3 not setup<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8458:0:(ldlm_request.c:1172:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Jul 2 16:10:17 wn47 kernel: Lustre: MGS has stopped.<br/> Jul 2 16:10:17 wn47 kernel: LustreError: 8458:0:(ldlm_request.c:1799:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Jul 2 16:10:23 wn47 kernel: Lustre: 8458:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1341238217/real 1341238217&#93;</span> req@ffff88041bdcac00 x1406387715309722/t0(0) o251-&gt;MGC10.10.65.47@tcp@0@lo:26/25 lens 192/192 e 0 to 1 dl 1341238223 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Jul 2 16:10:23 wn47 kernel: Lustre: server umount lustre-MDT0000 complete<br/> Jul 2 16:10:23 wn47 kernel: LustreError: 8458:0:(obd_mount.c:2198:lustre_fill_super()) Unable to mount (-22)<br/> Jul 2 16:10:23 wn47 kernel: LustreError: 8458:0:(obd_mount.c:2198:lustre_fill_super()) Skipped 1 previous similar message</p> <p>And attached you can find the lustre dk</p> <p>Fabio</p> <p>Please tell us exactly what you did, list all steps if possible.</p> <ul class="alternate" type="square"> <li>Did you download the 2.1.2 release from our site?</li> <li>Did you install the latest e2fsprogs, also from our site?</li> </ul> <p>Hi Fabio,</p> <p>As far as upgrade/downgrade is concerned, we only support the downgrade of a filesystem that was formatted with an "old" version of lustre. I mean that, you can format a filesystem with 2.x, upgrade it to 2.x+1 and then downgrade to 2.x. However, if you format with 2.x+1, we might enable new features (e.g. flex_bg is the first example i can think of) which won't be supported by prior releases. This has been a general rule with lustre releases for a while.</p> <p>In the present case, you can't mount the filesystem because of this message:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 2 16:10:17 wn47 kernel: LustreError: 8540:0:(mdt_recovery.c:409:mdt_server_data_init()) lustre-MDT0000: unsupported incompat filesystem feature(s) 200 </pre> </div></div> <p>The incompat feature is actually multiple object index support which was added in 2.2 (see <a href="https://jira.whamcloud.com/browse/LU-822" title="allow multiple Object Index files to be created" class="issue-link" data-issue-key="LU-822"><del>LU-822</del></a>).<br/> Because of this new feature which is enabled by default for any new filesystem formatted with 2.2, you can't downgrade to 2.1.</p> <p>HTH</p> <p>Johann,</p> <p>Unfortunately this latest piece of information (namely the fact that the filesystem needs to be reformatted because of incompatibilities between it and a v2.1.2) came too late for us to adequately inform our user community. The filesystem contained nearly 400TB of user data and we must allow at least 1 weeks advance notice so that users can move their crucial data to a more permanent filesystem. Consequently we will not be able to perform this downgrade until mid-July at the earliest. In fact we would prefer to wait until our next scheduled maintenance on Wednesday 8 August. This is a regrettable situation.</p> <p>On the plus side we have unmounted the second Lustre filesystem from the 1500 node XE6 (which was an additional complication) and this seems to have made the Lustre logs less "chatty" (i.e. not logging as many problems). The XE6 is however not completely full of jobs yet so we will keep a close eye on the situation over the coming days. Furthermore one of the OSSes needed a motherboard swap last week so it may have had a hardware fault that was also contributing to destabilise the filesystem.</p> <p>Finally FYI we have ordered 4 additional OSSes of identical spec to the ones we have in production. When they arrive (likely later in July) we will build a preproduction test environment (we have some unused DS4800 storage to use at the backend).</p> <p>Regards,</p> <p>Colin </p> <p>Thanks, please let us know what we can do to assist.</p> <p>I have two questions regarding the issue of the reconnecting compute nodes between Lustre &amp; Cray (gni interconnect), as my latest attached file we had this issue yesterday, the question is Lustre version 2.1.2 is running the same kind of code for connecting to gni that Lustre 2.2 is using? </p> <p>If the code is the same regarding the gni part will be really helpful to downgrade or we would experiencing the same issue?</p> <p>The second question is, is it possible that a tuning of the router/client nodes side, maybe increasing the timeout in the modprobe would help? </p> <p>Right now we have on the router node a timeout=250 but there is not any specification on the client side.</p> <p>Regards<br/> Fabio</p> <p>The gnilnd code is supplied by Cray - have you consulted them on this issue? We are at the present time unable to access the gnilnd source, so it is a bit difficult for us to say anything on this issue. We believe you would be better off in general running our current maintenance release (2.1.2) as you are currently hitting two known bugs that are not present in this release. Given your current state of Lustre experience 2.1.2 will be in general more stable. If you wish to continue with the feature release (2.2), you will need to patch your version of Lustre with the fixes for the bugs you have hit.</p> <p>Cray has been planning on pushing gnilnd upstream. We've opened <a href="https://jira.whamcloud.com/browse/LU-1419" title="Tracking ticket for gnilnd push" class="issue-link" data-issue-key="LU-1419"><del>LU-1419</del></a> to track that effort.</p> <p>For this ticket though, I think that it would help to be clear about what components and Lustre version you are discussing. It's my guess that this customer is still running 1.8.x clients on their Cray mainframe, and not version 2.2 or 2.1.2. I believe that in the course of this ticket, when 2.x is referenced, it refers to the server version only.</p> <p>BTW, the gnilnd has a feature to "quiesce" when the Cray high speed network is under a re-route for HW warmswap or link failure. If the Cray has gone quiescent, then you won't hear from its clients until the quiesce event is complete. During that time, messages to the Cray clients will probably even back up on the LNET routers such that all router buffers are consumed.</p> <p>I've just uploaded 2 new log files, related to a problem occurred yesterday, a lot of clients from our XE6 system lost connectivity with Lustre. </p> <p>I hope the logs help, log_11_jul is the one from the lustre servers (weisshorn07) the drop_conn is the log from Cray XE6, from 19:00 related to the error with weisshorn07.</p> <p>Tomorrow we are going to apply some tuning on Server, router &amp; client side, the following are the changes we are going to apply:</p> <ol> <li> <ol> <li>Router ##</li> </ol> </li> </ol> <p>options ko2iblnd timeout=100 peer_timeout=130<br/> options ko2iblnd credits=2048 ntx=2048<br/> options ko2iblnd peer_credits=126 concurrent_sends=63<br/> peer_buffer_credits=128<br/> options kgnilnd credits=2048 peer_health=1</p> <p>options lnet check_routers_before_use=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet router_ping_timeout=50<br/> options lnet large_router_buffers=1024 small_router_buffers=16384</p> <ol> <li> <ol> <li>Server ##</li> </ol> </li> </ol> <p>options ko2iblnd timeout=100 peer_timeout=0 keepalive=30<br/> options ko2iblnd credits=2048 ntx=2048<br/> options ko2iblnd peer_credits=126 concurrent_sends=63</p> <p>options lnet avoid_asym_router_failure=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet check_routers_before_use=1</p> <ol> <li> <ol> <li>Client ##</li> </ol> </li> </ol> <p>LOGIN - COMPUTE</p> <p>No need ko2ilbnd tuning on ROSA Login nodes because there are not IB cards.<br/> #options ko2iblnd timeout=100 peer_timeout=0 keepalive=30<br/> #options ko2iblnd credits=2048 ntx=2048<br/> #options ko2iblnd peer_credits=126 concurrent_sends=63</p> <p>options lnet avoid_asym_router_failure=1<br/> options lnet dead_router_check_interval=60<br/> options lnet live_router_check_interval=60<br/> options lnet check_routers_before_use=1</p> <ol> <li> <ol> <li>tuning ##</li> </ol> </li> </ol> <p>MDS</p> <p>echo "100" &gt; /proc/fs/lustre/lov/scratch-MDT0000-mdtlov/qos_threshold_rr <br/> echo "256" &gt; /proc/fs/lustre/mdt/scratch-MDT0000/mdt/threads_max<br/> lctl set_param lov.*.stripecount=4<br/> echo "240" &gt; /proc/fs/lustre/mdt/scratch-MDT0000/identity_acquire_expire</p> <p>OSS</p> <p>pdsh -a -x weisshorn<span class="error">&#91;01-02&#93;</span> "lctl set_param timeout=300"<br/> pdsh -a -x weisshorn<span class="error">&#91;01-02&#93;</span> "lctl set_param ldlm_timeout=230"<br/> pdsh -a -x weisshorn<span class="error">&#91;01-02&#93;</span> "lctl set_param at_min=230"</p> <p>Can you please confirm me that the qos_threshold is enough to tune only on the MDS and no need of changes on the client side?<br/> I noticed that the value on the clients is slightly different than the one on the server:</p> <p>fverzell@rosa5:~&gt; cat /proc/fs/lustre/lov/scratch-clilov-ffff8803fc233000/qos_threshold_rr <br/> 16%</p> <p><span class="error">&#91;root@weisshorn01 ~&#93;</span># cat /proc/fs/lustre/lov/scratch-MDT0000-mdtlov/qos_threshold_rr<br/> 17%</p> <p>Based on the fact that the FS seems to be unbalanced, it is necessary lfs_migrate to 'fix' the situation?</p> <p>fverzell@rosa5:~&gt; lfs df <br/> UUID 1K-blocks Used Available Use% Mounted on<br/> scratch-MDT0000_UUID 292202572 118544144 154174788 43% /scratch/weisshorn<span class="error">&#91;MDT:0&#93;</span><br/> scratch-OST0000_UUID 7512024488 4689026804 2447151520 66% /scratch/weisshorn<span class="error">&#91;OST:0&#93;</span><br/> scratch-OST0001_UUID 7512024488 4745725856 2390486948 67% /scratch/weisshorn<span class="error">&#91;OST:1&#93;</span><br/> scratch-OST0002_UUID 7512024488 5075884136 2060329692 71% /scratch/weisshorn<span class="error">&#91;OST:2&#93;</span><br/> scratch-OST0003_UUID 7512024488 4897558604 2238653176 69% /scratch/weisshorn<span class="error">&#91;OST:3&#93;</span><br/> scratch-OST0004_UUID 7512024488 4454906816 2681307012 62% /scratch/weisshorn<span class="error">&#91;OST:4&#93;</span><br/> scratch-OST0005_UUID 7512024488 4587438680 2548773100 64% /scratch/weisshorn<span class="error">&#91;OST:5&#93;</span><br/> scratch-OST0006_UUID 7512024488 3992329488 3143885212 56% /scratch/weisshorn<span class="error">&#91;OST:6&#93;</span><br/> scratch-OST0007_UUID 7512024488 4661958960 2474223000 65% /scratch/weisshorn<span class="error">&#91;OST:7&#93;</span><br/> scratch-OST0008_UUID 7512024488 4167201580 2969012160 58% /scratch/weisshorn<span class="error">&#91;OST:8&#93;</span><br/> scratch-OST0009_UUID 7512024488 4819927884 2316257380 68% /scratch/weisshorn<span class="error">&#91;OST:9&#93;</span><br/> scratch-OST000a_UUID 7512024488 4754286656 2381901152 67% /scratch/weisshorn<span class="error">&#91;OST:10&#93;</span><br/> scratch-OST000b_UUID 7512024488 4640072608 2496099572 65% /scratch/weisshorn<span class="error">&#91;OST:11&#93;</span><br/> scratch-OST000c_UUID 7512024488 4742974012 2393214728 66% /scratch/weisshorn<span class="error">&#91;OST:12&#93;</span><br/> scratch-OST000d_UUID 7512024488 2549614552 4586599524 36% /scratch/weisshorn<span class="error">&#91;OST:13&#93;</span><br/> scratch-OST000e_UUID 7512024488 4827107008 2309104772 68% /scratch/weisshorn<span class="error">&#91;OST:14&#93;</span><br/> scratch-OST000f_UUID 7512024488 4624081852 2512129928 65% /scratch/weisshorn<span class="error">&#91;OST:15&#93;</span><br/> scratch-OST0010_UUID 7512024488 4579714656 2556475388 64% /scratch/weisshorn<span class="error">&#91;OST:16&#93;</span><br/> scratch-OST0011_UUID 7512024488 4712685460 2423526320 66% /scratch/weisshorn<span class="error">&#91;OST:17&#93;</span><br/> scratch-OST0012_UUID 7512024488 4467238040 2668940296 63% /scratch/weisshorn<span class="error">&#91;OST:18&#93;</span><br/> scratch-OST0013_UUID 7512024488 3964966088 3171248460 56% /scratch/weisshorn<span class="error">&#91;OST:19&#93;</span><br/> scratch-OST0014_UUID 7512024488 4467610568 2668572328 63% /scratch/weisshorn<span class="error">&#91;OST:20&#93;</span><br/> scratch-OST0015_UUID 7512024488 4087648640 3048563952 57% /scratch/weisshorn<span class="error">&#91;OST:21&#93;</span><br/> scratch-OST0016_UUID 7512024488 2768510532 4367704256 39% /scratch/weisshorn<span class="error">&#91;OST:22&#93;</span><br/> scratch-OST0017_UUID 7512024488 4762759028 2373420232 67% /scratch/weisshorn<span class="error">&#91;OST:23&#93;</span><br/> scratch-OST0018_UUID 7512024488 4818779532 2317404560 68% /scratch/weisshorn<span class="error">&#91;OST:24&#93;</span><br/> scratch-OST0019_UUID 7512024488 2286805384 4849409388 32% /scratch/weisshorn<span class="error">&#91;OST:25&#93;</span><br/> scratch-OST001a_UUID 7512024488 4654062568 2482120052 65% /scratch/weisshorn<span class="error">&#91;OST:26&#93;</span><br/> scratch-OST001b_UUID 7512024488 4716531932 2419679848 66% /scratch/weisshorn<span class="error">&#91;OST:27&#93;</span><br/> scratch-OST001c_UUID 7512024488 4829306016 2306855832 68% /scratch/weisshorn<span class="error">&#91;OST:28&#93;</span><br/> scratch-OST001d_UUID 7512024488 4724954000 2411258804 66% /scratch/weisshorn<span class="error">&#91;OST:29&#93;</span><br/> scratch-OST001e_UUID 7512024488 4734034796 2402147544 66% /scratch/weisshorn<span class="error">&#91;OST:30&#93;</span><br/> scratch-OST001f_UUID 7512024488 4764160748 2372052056 67% /scratch/weisshorn<span class="error">&#91;OST:31&#93;</span><br/> scratch-OST0020_UUID 7512024488 4749913620 2386283024 67% /scratch/weisshorn<span class="error">&#91;OST:32&#93;</span><br/> scratch-OST0021_UUID 7512024488 4051468944 3084745524 57% /scratch/weisshorn<span class="error">&#91;OST:33&#93;</span><br/> scratch-OST0022_UUID 7512024488 2519479924 4616734864 35% /scratch/weisshorn<span class="error">&#91;OST:34&#93;</span><br/> scratch-OST0023_UUID 7512024488 5047050388 2089132448 71% /scratch/weisshorn<span class="error">&#91;OST:35&#93;</span><br/> scratch-OST0024_UUID 7512024488 4777844024 2358343904 67% /scratch/weisshorn<span class="error">&#91;OST:36&#93;</span><br/> scratch-OST0025_UUID 7512024488 2515885904 4620324508 35% /scratch/weisshorn<span class="error">&#91;OST:37&#93;</span><br/> scratch-OST0026_UUID 7512024488 4612974860 2523237944 65% /scratch/weisshorn<span class="error">&#91;OST:38&#93;</span><br/> scratch-OST0027_UUID 7512024488 4766933908 2369277872 67% /scratch/weisshorn<span class="error">&#91;OST:39&#93;</span><br/> scratch-OST0028_UUID 7512024488 4732878372 2403333408 66% /scratch/weisshorn<span class="error">&#91;OST:40&#93;</span><br/> scratch-OST0029_UUID 7512024488 4805835488 2330355492 67% /scratch/weisshorn<span class="error">&#91;OST:41&#93;</span><br/> scratch-OST002a_UUID 7512024488 4623393332 2512795728 65% /scratch/weisshorn<span class="error">&#91;OST:42&#93;</span><br/> scratch-OST002b_UUID 7512024488 4817231168 2318980612 68% /scratch/weisshorn<span class="error">&#91;OST:43&#93;</span><br/> scratch-OST002c_UUID 7512024488 2429377284 4706837500 34% /scratch/weisshorn<span class="error">&#91;OST:44&#93;</span><br/> scratch-OST002d_UUID 7512024488 4825606516 2310560384 68% /scratch/weisshorn<span class="error">&#91;OST:45&#93;</span><br/> scratch-OST002e_UUID 7512024488 2666838228 4469376560 37% /scratch/weisshorn<span class="error">&#91;OST:46&#93;</span><br/> scratch-OST002f_UUID 7512024488 3925127408 3211050572 55% /scratch/weisshorn<span class="error">&#91;OST:47&#93;</span><br/> scratch-OST0030_UUID 7512024488 4727638656 2408540452 66% /scratch/weisshorn<span class="error">&#91;OST:48&#93;</span><br/> scratch-OST0031_UUID 7512024488 2693940128 4442271700 38% /scratch/weisshorn<span class="error">&#91;OST:49&#93;</span><br/> scratch-OST0032_UUID 7512024488 4764748760 2371463020 67% /scratch/weisshorn<span class="error">&#91;OST:50&#93;</span><br/> scratch-OST0033_UUID 7512024488 4864564316 2271648488 68% /scratch/weisshorn<span class="error">&#91;OST:51&#93;</span><br/> scratch-OST0034_UUID 7512024488 4625740760 2510443620 65% /scratch/weisshorn<span class="error">&#91;OST:52&#93;</span><br/> scratch-OST0035_UUID 7512024488 4693269256 2442910852 66% /scratch/weisshorn<span class="error">&#91;OST:53&#93;</span><br/> scratch-OST0036_UUID 7512024488 4544146008 2592030580 64% /scratch/weisshorn<span class="error">&#91;OST:54&#93;</span><br/> scratch-OST0037_UUID 7512024488 4860051744 2276133072 68% /scratch/weisshorn<span class="error">&#91;OST:55&#93;</span><br/> scratch-OST0038_UUID 7512024488 2854611264 4281603524 40% /scratch/weisshorn<span class="error">&#91;OST:56&#93;</span><br/> scratch-OST0039_UUID 7512024488 4750769272 2385415720 67% /scratch/weisshorn<span class="error">&#91;OST:57&#93;</span><br/> scratch-OST003a_UUID 7512024488 2387156708 4749058080 33% /scratch/weisshorn<span class="error">&#91;OST:58&#93;</span><br/> scratch-OST003b_UUID 7512024488 4971879748 2164295048 70% /scratch/weisshorn<span class="error">&#91;OST:59&#93;</span><br/> scratch-OST003c_UUID 7512024488 4768532532 2367679248 67% /scratch/weisshorn<span class="error">&#91;OST:60&#93;</span><br/> scratch-OST003d_UUID 7512024488 2486278472 4649936292 35% /scratch/weisshorn<span class="error">&#91;OST:61&#93;</span><br/> scratch-OST003e_UUID 7512024488 4799709104 2336473268 67% /scratch/weisshorn<span class="error">&#91;OST:62&#93;</span><br/> scratch-OST003f_UUID 7512024488 4804003128 2332208652 67% /scratch/weisshorn<span class="error">&#91;OST:63&#93;</span><br/> scratch-OST0040_UUID 7512024488 4737174800 2399036980 66% /scratch/weisshorn<span class="error">&#91;OST:64&#93;</span><br/> scratch-OST0041_UUID 7512024488 4570719888 2565491892 64% /scratch/weisshorn<span class="error">&#91;OST:65&#93;</span><br/> scratch-OST0042_UUID 7512024488 4680780720 2455398272 66% /scratch/weisshorn<span class="error">&#91;OST:66&#93;</span><br/> scratch-OST0043_UUID 7512024488 4876523204 2259658024 68% /scratch/weisshorn<span class="error">&#91;OST:67&#93;</span><br/> scratch-OST0044_UUID 7512024488 4016775868 3119437956 56% /scratch/weisshorn<span class="error">&#91;OST:68&#93;</span><br/> scratch-OST0045_UUID 7512024488 4909041500 2227172328 69% /scratch/weisshorn<span class="error">&#91;OST:69&#93;</span><br/> scratch-OST0046_UUID 7512024488 4894087388 2242097828 69% /scratch/weisshorn<span class="error">&#91;OST:70&#93;</span><br/> scratch-OST0047_UUID 7512024488 4750555368 2385615184 67% /scratch/weisshorn<span class="error">&#91;OST:71&#93;</span></p> <p>Thanks<br/> Fabio</p> <p>this is the second option of tuning to apply:</p> <ol> <li> <ol> <li>Router nodes ##</li> </ol> </li> </ol> <p>options mlx4_core msi_x=1</p> <p>options kgnilnd credits=4000 timeout=120</p> <p>options qla2xxx ql2xlogintimeout=0<br/> options ost oss_num_threads=256<br/> options libcfs libcfs_panic_on_lbug=1</p> <p>options ko2iblnd credits=1024 ntx=2048 </p> <p>options lnet small_router_buffers=16384<br/> ############################################</p> <ol> <li> <ol> <li>compute nodes ##</li> </ol> </li> </ol> <p>options lnet check_routers_before_use=1<br/> options lnet dead_router_check_interval=150<br/> options lnet live_router_check_interval=150<br/> options lnet router_ping_timeout=130<br/> options kgnilnd credits=1024 timeout=120<br/> ############################################</p> <ol> <li> <ol> <li>Servers ##</li> </ol> </li> </ol> <p>options mlx4_core msi_x=1<br/> options lnet networks=o2ib(ib0)</p> <p>options lnet check_routers_before_use=1<br/> options lnet router_ping_timeout=130<br/> options lnet dead_router_check_interval=150<br/> options lnet live_router_check_interval=150</p> <p>options ko2iblnd credits=1024 ntx=2048 peer_credits=8</p> <p>options libcfs libcfs_panic_on_lbug=1<br/> ############################################</p> <p>There's any pro &amp; cons at a first glance about the first &amp; the second set of tuning we want to try tomorrow?</p> <p>Fabio</p> <p>The filesystem does not seem especially unbalanced, lfs migrate is used when adding new storage. What exactly is your concern? Normal usage should balance the space out, depending on your mix of workloads. it looks like your average OST is 66% full, with a few at ~30%. That should balance out in normal usage.</p> <p>And yes, QOS parameters are set on the MDS, as MDS controls that allocation.</p> <p>Is there anything more we can do on this issue, or is it okay to close?</p> Development Rank 1|hzv3cn: Rank (Obsolete) 4042 Severity