https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13632 Lustre <p>One of our Lustre storage servers has become unstable. Below are the messages we found on the 2 OSS that fell over (rebooted) due to bad write checksums. We don't recall seeing this before. We tracked down the files referenced via the Hex codes in the error messages below with the lfs fid2path command and killed the user's jobs. I will mentioned that we did update our lnet routers from running 2.13.0 to 2.12.4 in the past week or two. Also the lustre clients that are accessing our Lustre storage are running lustre 2.10. 7-1.</p> <p>This particular lustre server that is unstable is running lustre 2.12.3:<br/> <span class="error">&#91;root@holyscratch01mds01 ~&#93;</span># rpm -qa |grep lustre<br/> kernel-3.10.0-1062.1.1.el7_lustre.x86_64<br/> kmod-lustre-2.12.3-1.el7.x86_64<br/> kmod-lustre-osd-ldiskfs-2.12.3-1.el7.x86_64<br/> kernel-devel-3.10.0-1062.1.1.el7_lustre.x86_64<br/> lustre-osd-zfs-mount-2.12.3-1.el7.x86_64<br/> lustre-2.12.3-1.el7.x86_64<br/> lustre-zfs-dkms-2.12.3-1.el7.noarch<br/> lustre-resource-agents-2.12.3-1.el7.x86_64<br/> lustre-ldiskfs-zfs-5.0.0-1.el7.x86_64<br/> kernel-mft-4.13.3-3.10.0_1062.1.1.el7_lustre.x86_64.x86_64<br/> lustre-osd-ldiskfs-mount-2.12.3-1.el7.x86_64<br/> kmod-spl-3.10.0-1062.1.1.el7_lustre.x86_64-0.7.13-1.el7.x86_64</p> <p>Here are the errors:</p> <p>Jun 3 18:23:54 holyscratch01oss03 kernel: LustreError: 168-f: scratch1-OST001a: BAD WRITE CHECKSUM: from 12345-10.31.164.172@o2ib via 10.31.179.131@o2ib4 inode <span class="error">&#91;0x20001bc69:0xa3f6:0x0&#93;</span> object 0x0:76971842 extent <span class="error">&#91;71303168-75497471&#93;</span>: client csum e6b01811, server csum 880d3728<br/> Jun 3 18:23:55 holyscratch01oss03 kernel: LustreError: 168-f: scratch1-OST0009: BAD WRITE CHECKSUM: from 12345-10.31.163.222@o2ib via 10.31.179.133@o2ib4 inode <span class="error">&#91;0x200012232:0xb3f4:0x0&#93;</span> object 0x0:76503185 extent <span class="error">&#91;1279262720-1283457023&#93;</span>: client csum e611ac34, server csum 82fdc50a</p> <p>Any ideas as to why this happening ?</p> Server side is all Dell gear. MDS and OSS are R740, storage is Dell ME4 storage enclosures. lnet routers are Lenovo SR630 running lustre 2.12.4, clients are mostly Dell and Lenovo gear running 2.10.7-1. LU-13632

BAD WRITE CHECKSUM

Bug Critical Open Unresolved Wang Shilong Michael Ethier Thu, 4 Jun 2020 00:29:08 +0000 Thu, 23 Jul 2020 19:47:54 +0000 Lustre 2.12.3 0 10 <p>Michael</p> <p>You have opened this ticket as severity 1 which means that your whole filesystem is out of services - is that the case? From the description it sounds like you are in production but want to root cause what has happened to the impacted files.</p> <p>Peter</p> <p>Hi Peter,<br/> Its has gone up and down a least 2 times since 5pm today. We had it running but shortly after we lose 2 OSS. So basically its not stable and useable at this point. So its not completely down. Sorry for not selecting the correct value. Your help page didn't have info on that selection when I looked.<br/> Mike</p> <p>We are seeing errors like this on an OST, <span class="image-wrap" style=""><a id="35053_thumb" href="https://jira.whamcloud.com/secure/attachment/35053/35053_Screen+Shot+2020-06-03+at+9.24.17+PM.png" title="Screen Shot 2020-06-03 at 9.24.17 PM.png" file-preview-type="image" file-preview-id="35053" file-preview-title="Screen Shot 2020-06-03 at 9.24.17 PM.png"><img src="https://jira.whamcloud.com/secure/thumbnail/35053/_thumb_35053.png" style="border: 0px solid black" role="presentation"/></a></span> see attached.</p> <p>No problem Michael - I just wanted to be clear as to what the priority was at this point. It sounds like getting things stable is the first priority. </p> <p>Shilong - could you please advise?</p> <p>Also we continue to have issues with OSTs with these errors and that OSTs won’t stay mounted, and this is causing OSS to consistently reboot</p> <p>Update: we haven’t had an OST failover in about 30min so it seems stable, but we have OSTs that won’t fail back without failure.</p> <p>Michael,</p> <p>can you run</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl get_param osc.*.checksum_type </pre> </div></div> <p>on one of the clients seeing the error, e.g. 10.31.164.172 or 10.31.163.222</p> <p>and gather the output?</p> <p>Thanks</p> <p>Hi Dongyang,<br/> The output of that command is attached.<br/> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/35060/35060_holy2b11102.out" title="holy2b11102.out attached to LU-13632">holy2b11102.out^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mre64" class="user-hover" rel="mre64">mre64</a>,</p> <p>Are the clients, LNet routers, and servers all using ECC memory? Are there any memory errors in the logs on the path from client to server? It would also be good to rule out any network hardware errors?</p> <p>Hi John,<br/> Yes they are using ECC memory. I checked the server side Lustre nodes, the lnet routers, and the client nodes that were referenced by the BAD WRITE CHECKSUM and none of them have memory errors.</p> <p>Thanks Michael.</p> <p>Some questions to help us isolate this:</p> <ol> <li>Are the checksum errors associated with multiple applications?</li> <li>Did they start soon after the router upgrade?</li> <li>Do they only occur on routed clients?</li> <li>Would it be possible to revert the router upgrade and see if the checksum errors stop for RPCs that use the downgraded router?</li> <li>Would it be possible to bring some clients to 2.12.4 and see if they occur?</li> </ol> <p>Hi John,</p> <p>1. It seems the CHECKSUM error was coming from the same type of application and user. However, that user has been running the same type of jobs for months on the same group of nodes.<br/> 2. We have been changing the lnet routers to be exactly the same for past 2-3 weeks, so they were not done all within a short period of time.<br/> 3. We saw the BAD WRITE CHECKSUM on both routed and directly connected (via HDR IB) nodes.<br/> 4. I don't think we are in the position to change the lnet routers back to 2.13.0. Also since we saw the CHECKSUM error coming from nodes that are not using the lnet routers, it seems the routers are not the problem.<br/> 5. We are running a specific kernel and OS so it maybe possible to update the client on some of the compute nodes if 2.12.4 installs without issue or errors. Our compute nodes are running Centos 7.6.1810 3.10.0-957.12.1.el7.x86_64 kernel with OFED INBOX drivers, not MLNX OFED. We would rather not do this to be honest.</p> <p>Thanks Michael,</p> <p>from the output we can see the checksum type is crc32c:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> osc.scratch1-OST0009-osc-ffff9ff17a698000.checksum_type=crc32 adler [crc32c] osc.scratch1-OST001a-osc-ffff9ff17a698000.checksum_type=crc32 adler [crc32c] </pre> </div></div> <p>You mentioned the clients are using OFED inbox drivers,</p> <p>are they always using inbox drivers? and the version of it?</p> <p>what about the lnet router and oss servers? Are they using MOFED and do we have the same version as the clients?</p> <p>Hi Dongyang,</p> <p>The compute clients have always been using the OFED INBOX drivers, for example:</p> <p><span class="error">&#91;root@holy7c02108 ~&#93;</span># modinfo mlx5_ib<br/> filename: /lib/modules/3.10.0-957.12.1.el7.x86_64/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko.xz<br/> license: Dual BSD/GPL<br/> description: Mellanox Connect-IB HCA IB driver<br/> author: Eli Cohen &lt;eli@mellanox.com&gt;<br/> retpoline: Y<br/> rhelversion: 7.6<br/> srcversion: 3B27ACD7C17E508C4D27B18<br/> depends: mlx5_core,ib_core<br/> intree: Y<br/> vermagic: 3.10.0-957.12.1.el7.x86_64 SMP mod_unload modversions <br/> signer: CentOS Linux kernel signing key<br/> sig_key: 2C:7C:17:70:5C:86:D4:20:80:50:D3:F5:54:56:9A:7B:D3:BF:D1:BF<br/> sig_hashalgo: sha256</p> <p>The lnet routers are using MLNX_OFED_LINUX-4.7-1.0.0.1 (OFED-4.7-1.0.0).</p> <p>The lustre server is using 4.7.1.0.0.1:<br/> -bash-4.2$ rpm -qa |grep -i mlnx<br/> mlnx-ofa_kernel-4.7-OFED.4.7.1.0.0.1.g1c4bf42.x86_64<br/> libibumad-static-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64<br/> mlnx-ofa_kernel-devel-4.7-OFED.4.7.1.0.0.1.g1c4bf42.x86_64<br/> ibutils2-2.1.1-0.113.MLNX20191121.g1c29603.47329.x86_64<br/> libibumad-devel-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64<br/> libibumad-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64<br/> kmod-mlnx-ofa_kernel-4.7-OFED.4.7.1.0.0.1.g1c4bf42.x86_64</p> <p>Also, the lnet routers have the MLNX OFED "upstream" drivers which are the latest/greatest while it appears the server side is using the MLNX legacy driver:</p> <p>Server side:<br/> -bash-4.2$ modinfo mlx5_ib<br/> filename: /lib/modules/3.10.0-1062.1.1.el7_lustre.x86_64/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko<br/> license: Dual BSD/GPL<br/> description: Mellanox Connect-IB HCA IB driver<br/> author: Eli Cohen &lt;eli@mellanox.com&gt;<br/> retpoline: Y<br/> rhelversion: 7.7<br/> srcversion: 706E928F4D4ECF2659B961F<br/> depends: mlx5_core,ib_core,ib_uverbs,mlx_compat<br/> vermagic: 3.10.0-1062.1.1.el7_lustre.x86_64 SMP mod_unload modversions <br/> parm: dc_cnak_qp_depth:DC CNAK QP depth (uint)<br/> -bash-4.2$ rpm -qf /lib/modules/3.10.0-1062.1.1.el7_lustre.x86_64/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko<br/> kmod-mlnx-ofa_kernel-4.7-OFED.4.7.1.0.0.1.g1c4bf42.x86_64</p> <p>However, like I mentioned before we have some compute nodes that are connected to the same HDR fabric as the lustre server (holyscratch01), and don't go thru the lnet routers. Those nodes were also giving the BAD WRITE CHECKSUM error.</p> <p>I would like to mention I'm seeing a lot of these server_bulk_callback errors every 3-5 sec on oss03 and also I have seen these errors on oss06 (which has been flakey lately) which we ended up failing all of oss06 OSTs to oss05 because of its instability:</p> <p>Jun 4 21:49:48 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ecbd649ae00<br/> Jun 4 21:49:48 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec19c3c1000<br/> Jun 4 21:49:48 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9eb2c6e59000<br/> Jun 4 21:49:52 holyscratch01oss03 kernel: LustreError: 10895:0:(ldlm_lib.c:3262:target_bulk_io()) @@@ network error on bulk READ req@ffff9ea2b5496050 x1661185873568080/t0(0) o3-&gt;64c2d217-765e-c934-68ad-1480c3b9eac2@10.31.130.246@tcp:2/0 lens 608/440 e 0 to 0 dl 1591321807 ref 1 fl Interpret:/0/0 rc 0/0<br/> Jun 4 21:49:52 holyscratch01oss03 kernel: LustreError: 10895:0:(ldlm_lib.c:3262:target_bulk_io()) Skipped 134 previous similar messages<br/> Jun 4 21:49:54 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec449cf6c00<br/> Jun 4 21:50:00 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec19c3c2400<br/> Jun 4 21:50:00 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ef420aeb200<br/> Jun 4 21:50:07 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9eb2c6e58600<br/> Jun 4 21:50:07 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec449cf0600<br/> Jun 4 21:50:13 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec31bdc2c00<br/> Jun 4 21:50:19 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ebbd9c8f200<br/> Jun 4 21:50:19 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ef075f1a800<br/> Jun 4 21:50:26 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ec54f615a00<br/> Jun 4 21:50:32 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ecbd6499000<br/> Jun 4 21:50:38 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9e9ac410d600<br/> Jun 4 21:50:38 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9e98ca39bc00<br/> Jun 4 21:50:43 holyscratch01oss03 systemd: Starting IML Swap Emitter...<br/> Jun 4 21:50:43 holyscratch01oss03 systemd: Started IML Swap Emitter.<br/> Jun 4 21:50:45 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9eb49b8b9a00<br/> Jun 4 21:50:57 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ecd6d92ba00<br/> Jun 4 21:51:04 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff9ee813e90000<br/> Jun 4 21:51:04 holyscratch01oss03 kernel: Lustre: scratch1-OST0014: Bulk IO write error with a10910a5-f0a8-e30e-b540-a728a40bd183 (at 10.31.163.219@o2ib), client will retry: rc = -110<br/> Jun 4 21:51:04 holyscratch01oss03 kernel: Lustre: Skipped 73 previous similar messages<br/> Jun 4 21:51:07 holyscratch01oss03 kernel: LustreError: 11313:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304) req@ffff9eaf5d2a8850 x1644618841213632/t0(0) o4-&gt;a5f73f03-2a7d-38af-559e-bc8ad5f84416@10.31.167.138@o2ib:182/0 lens 608/448 e 0 to 0 dl 1591321987 ref 1 fl Interpret:/0/0 rc 0/0<br/> Jun 4 21:51:07 holyscratch01oss03 kernel: LustreError: 11313:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 68 previous similar messages<br/> Jun 4 21:51:10 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9eb161ddb200<br/> Jun 4 21:51:10 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ee83c77a600<br/> Jun 4 21:51:10 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ebbd9c8f400<br/> Jun 4 21:51:23 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ebf48b49a00<br/> Jun 4 21:51:23 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ecc98b41c00<br/> Jun 4 21:51:29 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9edc45d7bc00<br/> Jun 4 21:51:29 holyscratch01oss03 kernel: LustreError: 8683:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9ee83c77e800</p> <p>Michael,</p> <p>can you create a file using lfs setstripe to create the stripes on the osts running on oss03(or whichever showing up the errors),</p> <p>and do a simple dd(both read and write) from the client and see if you are having the checksum errors?</p> <p>Hi Dongyang,<br/> I tried that and I am not seeing any checksum errors on the oss03 /var/log/messages file when I run the dd.</p> <p>Hi Dongyang, this is what I did exactly:</p> <p>local OSTs on oss03:<br/> <span class="error">&#91;root@holyscratch01oss03 ~&#93;</span># df -hl<br/> Filesystem Size Used Avail Use% Mounted on<br/> devtmpfs 189G 0 189G 0% /dev<br/> tmpfs 189G 39M 189G 1% /dev/shm<br/> tmpfs 189G 43M 189G 1% /run<br/> tmpfs 189G 0 189G 0% /sys/fs/cgroup<br/> /dev/mapper/centos_holyscratch01oss03-root 218G 7.7G 211G 4% /<br/> /dev/sda1 1014M 200M 815M 20% /boot<br/> /dev/mapper/mpathg 85T 46T 35T 57% /mnt/scratch1-OST0014<br/> /dev/mapper/mpathe 85T 45T 36T 56% /mnt/scratch1-OST000e<br/> /dev/mapper/mpathi 85T 44T 37T 55% /mnt/scratch1-OST0002<br/> /dev/mapper/mpathj 85T 44T 38T 54% /mnt/scratch1-OST0008<br/> /dev/mapper/mpathm 85T 47T 35T 58% /mnt/scratch1-OST001a<br/> /dev/mapper/mpathc 85T 47T 35T 58% /mnt/scratch1-OST002c<br/> /dev/mapper/mpathn 85T 44T 37T 55% /mnt/scratch1-OST0020<br/> /dev/mapper/mpatha 85T 45T 36T 56% /mnt/scratch1-OST0026</p> <p>Lists the OSTs on the holyscatch01 lustre FS:</p> <p><span class="error">&#91;root@holy2c24108 ~&#93;</span># lfs osts /n/holyscratch01<br/> OBDS:<br/> 0: scratch1-OST0000_UUID ACTIVE<br/> 1: scratch1-OST0001_UUID ACTIVE<br/> 2: scratch1-OST0002_UUID ACTIVE<br/> 3: scratch1-OST0003_UUID ACTIVE<br/> 4: scratch1-OST0004_UUID ACTIVE<br/> 5: scratch1-OST0005_UUID ACTIVE<br/> 6: scratch1-OST0006_UUID ACTIVE<br/> 7: scratch1-OST0007_UUID ACTIVE<br/> 8: scratch1-OST0008_UUID ACTIVE<br/> 9: scratch1-OST0009_UUID ACTIVE<br/> 10: scratch1-OST000a_UUID ACTIVE<br/> 11: scratch1-OST000b_UUID ACTIVE<br/> 12: scratch1-OST000c_UUID ACTIVE<br/> 13: scratch1-OST000d_UUID ACTIVE<br/> 14: scratch1-OST000e_UUID ACTIVE<br/> 15: scratch1-OST000f_UUID ACTIVE<br/> 16: scratch1-OST0010_UUID ACTIVE<br/> 17: scratch1-OST0011_UUID ACTIVE<br/> 18: scratch1-OST0012_UUID ACTIVE<br/> 19: scratch1-OST0013_UUID ACTIVE<br/> 20: scratch1-OST0014_UUID ACTIVE<br/> 21: scratch1-OST0015_UUID ACTIVE<br/> 22: scratch1-OST0016_UUID ACTIVE<br/> 23: scratch1-OST0017_UUID ACTIVE<br/> 24: scratch1-OST0018_UUID ACTIVE<br/> 25: scratch1-OST0019_UUID ACTIVE<br/> 26: scratch1-OST001a_UUID ACTIVE<br/> 27: scratch1-OST001b_UUID ACTIVE<br/> 28: scratch1-OST001c_UUID ACTIVE<br/> 29: scratch1-OST001d_UUID ACTIVE<br/> 30: scratch1-OST001e_UUID ACTIVE<br/> 31: scratch1-OST001f_UUID ACTIVE<br/> 32: scratch1-OST0020_UUID ACTIVE<br/> 33: scratch1-OST0021_UUID ACTIVE<br/> 34: scratch1-OST0022_UUID ACTIVE<br/> 35: scratch1-OST0023_UUID ACTIVE<br/> 36: scratch1-OST0024_UUID ACTIVE<br/> 37: scratch1-OST0025_UUID ACTIVE<br/> 38: scratch1-OST0026_UUID ACTIVE<br/> 39: scratch1-OST0027_UUID ACTIVE<br/> 40: scratch1-OST0028_UUID ACTIVE<br/> 41: scratch1-OST0029_UUID ACTIVE<br/> 42: scratch1-OST002a_UUID ACTIVE<br/> 43: scratch1-OST002b_UUID ACTIVE<br/> 44: scratch1-OST002c_UUID ACTIVE<br/> 45: scratch1-OST002d_UUID ACTIVE<br/> 46: scratch1-OST002e_UUID ACTIVE<br/> 47: scratch1-OST002f_UUID ACTIVE</p> <p>Execute the lfs setstripe:<br/> lfs setstripe --ost-list 2,8,14,20,26,32,38,44 /n/holyscratch01/rc_admin/methier/teststripe</p> <p>The dd write:<br/> <span class="error">&#91;root@holy2c24108 ~&#93;</span># dd if=/dev/urandom of=/n/holyscratch01/rc_admin/methier/teststripe count=1024 bs=10M</p> <p>The read I just swapped if with of:<br/> <span class="error">&#91;root@holy2c24108 ~&#93;</span># dd of=/dev/urandom if=/n/holyscratch01/rc_admin/methier/teststripe count=1024 bs=10M</p> <p><span class="error">&#91;root@holy2c24108 ~&#93;</span># lfs getstripe /n/holyscratch01/rc_admin/methier/teststripe<br/> /n/holyscratch01/rc_admin/methier/teststripe<br/> lmm_stripe_count: 8<br/> lmm_stripe_size: 1048576<br/> lmm_pattern: 1<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: 2<br/> obdidx objid objid group<br/> 2 78461277 0x4ad395d 0<br/> 8 79385285 0x4bb52c5 0<br/> 14 77802814 0x4a32d3e 0<br/> 20 77643661 0x4a0bf8d 0<br/> 26 77265238 0x49af956 0<br/> 32 79283352 0x4b9c498 0<br/> 38 78144208 0x4a862d0 0<br/> 44 77354743 0x49c56f7 0</p> <p>Just to clarify, you are still running 2.12.3 on the servers, and not 2.12.4? There was a known issue with 2.12.3 (<a href="https://jira.whamcloud.com/browse/LU-13020" title="ko2iblnd tuning" class="issue-link" data-issue-key="LU-13020"><del>LU-13020</del></a>, <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a>) that affected LNet under load on larger system that may be contributing to the problem here. It appears that you are seeing the checksum errors because the bulk data transfers are being interrupted. </p> <p>A workaround to get equivalent behavior for 2.12.4 systems without upgrading or applying a patch is to run the following commands on all of the 2.12.3 nodes in the shown order:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo 150 &gt; /sys/module/lnet/parameters/lnet_transaction_timeout echo 3 &gt; /sys/module/lnet/parameters/lnet_retry_count </pre> </div></div> <p>This only temporarily changes these values, but they can be set permanently by adding the following line in <tt>/etc/modprobe.d/lnet.conf</tt> on all 2.13.3 nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options lnet lnet_retry_count=3 lnet_transaction_timeout=150 </pre> </div></div> <p>Another thing to try to reduce the severity of the problem, if the above does not help would be setting "<tt>lctl set_param osc.&#42;.max_pages_per_rpc=1M</tt>" on the clients, which will reduce the number of bulk transfers per RPC, which should at least avoid the checksum errors being reported, and avoid network traffic congestion if there <b>are</b> still transfer errors, since smaller RPCs would also need less resent each time upon hitting an error. This would only affect the current clients, so you could set "<tt>lctl set_param obdfilter.&#42;.brw_size=1</tt>" on all OSS nodes to limit this for future client mounts as well. </p> <p>Hi Andreas,<br/> Thanks for the useful info. Below is what we have set on the lnet routers, do you see an issue with these settings ?<br/> Thanks,<br/> Mike</p> <p><span class="error">&#91;root@cannonlnet06 ~&#93;</span># more /etc/modprobe.d/lustre.conf <br/> options lnet networks="o2ib(ib1),o2ib2(ib1),o2ib4(ib0),tcp(bond0),tcp4(bond0.2475)"<br/> options lnet forwarding="enabled"<br/> options lnet lnet_peer_discovery_disabled=1<br/> options lnet lnet_health_sensitivity=0</p> <p><span class="error">&#91;root@cannonlnet06 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 0<br/> transaction_timeout: 50<br/> health_sensitivity: 0<br/> recovery_interval: 1</p> <p>Also the lustre FS we have checksum errors on is lustre 2.12.3, yes. We have another lustre fs nearby running 2.12.4 and we have not seen checksum errors or instability on it recently. Its not getting hit with as much file I/O most likely as its a lab data storage server. This scratch01 (where checksums errors occur) server gets hammered by 2000 or so compute nodes as a temporary disk space to write/read their compute output.</p> <p>Serguei, could you review the LNet parameters in <a href="#comment-272090" target="_blank" rel="noopener">comment-272090</a>.</p> <p>I would recommend also trying the newer 2.12.4 clients on at least some of the clients to determine if this resolves the issue.</p> <p>Hi,</p> <p>LNet parameters look fine to me. Perhaps <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ashehata" class="user-hover" rel="ashehata">ashehata</a> can take a quick look to double-check. Because there appear to be multiple interfaces of the same kind, I'd also recommend checking if Linux routing is setup as outlined here: <a href="http://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node." class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node.</a></p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Andreas,<br/> I tried setting:<br/> echo 150 &gt; /sys/module/lnet/parameters/lnet_transaction_timeout<br/> echo 3 &gt; /sys/module/lnet/parameters/lnet_retry_count</p> <p> But its not letting me change retry_count:<br/> <span class="error">&#91;root@salt ~&#93;</span># salt 'holyscratch01*' cmd.run "echo 3 &gt; /sys/module/lnet/parameters/lnet_retry_count"<br/> holyscratch01oss01.rc.fas.harvard.edu:<br/> /bin/sh: line 0: echo: write error: Invalid argument<br/> holyscratch01oss06.rc.fas.harvard.edu:<br/> /bin/sh: /sys/module/lnet/parameters/lnet_retry_count: No such file or directory<br/> holyscratch01oss04.rc.fas.harvard.edu:<br/> /bin/sh: line 0: echo: write error: Invalid argument<br/> holyscratch01oss02.rc.fas.harvard.edu:</p> <p>holyscratch01oss03.rc.fas.harvard.edu:<br/> /bin/sh: /sys/module/lnet/parameters/lnet_retry_count: No such file or directory<br/> holyscratch01mds02.rc.fas.harvard.edu:<br/> /bin/sh: line 0: echo: write error: Invalid argument<br/> holyscratch01oss05.rc.fas.harvard.edu:<br/> /bin/sh: line 0: echo: write error: Invalid argument<br/> holyscratch01mds01.rc.fas.harvard.edu:</p> <p>What do you recommend ? Is the order correct ? I change the value transaction_timeout to 150 and tried to change it after that to a lower value and it doesn't let me now. There must be some kind of dependency between these two parameters that doesn't allow you to change them.</p> <p>Thanks,<br/> Mike</p> <p><span class="error">&#91;root@holyscratch01oss04 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 0<br/> transaction_timeout: 150<br/> health_sensitivity: 0<br/> recovery_interval: 1<br/> <span class="error">&#91;root@holyscratch01oss04 ~&#93;</span># lnetctl set retry_count 3<br/> add:</p> <ul class="alternate" type="square"> <li>retry_count:<br/> errno: -5<br/> descr: "cannot configure retry count: Invalid argument"</li> </ul> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mre64" class="user-hover" rel="mre64">mre64</a>,</p> <p>If you have health_sensitivity set to 0, you are prevented from setting non-zero retry_count as the health feature is off. You should still be able to change transaction_timeout though.</p> <p>Thanks,</p> <p>Serguei</p> <p>Hi Serguei,<br/> Thanks. So in our case, setting lnet_transaction_timeout=150 will be the only setting we should change, per Andreas suggestion ? Does that mean that if retry_county is set to 3 or some other value, its ignored when health is turned off ?<br/> Mike</p> <p>Right, if <tt>health_sensitivity=</tt> then these parameters are ignored, which is fine. It means that LNet will not interrupt incomplete RPCs to retry sending them to the server. Sorry, I didn't realize this was the case.</p> <p>Hi Andreas, I went ahead and set this on all the hosts that make up the scratch01 filesystem in /etc/modprobe.d/lustre.conf:<br/> options lnet lnet_health_sensitivity=100 lnet_retry_count=3 lnet_transaction_timeout=150<br/> The reason we had health turned off is it seemed to cause issues for us in the past. Do you recommend we turn health on or off ?<br/> Mike</p> <p>Hi Andreas,</p> <p>I set the following settings, and we still have the Bulk IO and server_bulk_callback messages on 2 of the OSS machines:</p> <p>options lnet lnet_health_sensitivity=100 lnet_retry_count=3 lnet_transaction_timeout=150 lnet_peer_discovery_disabled=1</p> <p><span class="error">&#91;root@holyscratch01oss06 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 3<br/> transaction_timeout: 150<br/> health_sensitivity: 100<br/> recovery_interval: 1</p> <p>Also checked the lctl get_param osc.*.max_pages_per_rpc from one of the clients the gives the bulk I/O and callback messages and its all 1M:<br/> <span class="error">&#91;root@holy2c18214 ~&#93;</span># lctl get_param osc.*.max_pages_per_rpc |grep scratch<br/> osc.scratch1-OST0000-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0001-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0002-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0003-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0004-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0005-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0006-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0007-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0008-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0009-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000a-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000b-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000c-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000d-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000e-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST000f-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0010-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0011-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0012-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0013-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0014-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0015-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0016-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0017-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0018-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0019-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001a-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001b-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001c-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001d-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001e-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST001f-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0020-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0021-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0022-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0023-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0024-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0025-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0026-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0027-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0028-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST0029-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002a-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002b-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002c-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002d-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002e-osc-ffff957cc6b6c800.max_pages_per_rpc=1024<br/> osc.scratch1-OST002f-osc-ffff957cc6b6c800.max_pages_per_rpc=1024</p> <p>We are going to see if we can update the lustre client to 2.12.4. It does build on our older centos 7.6 OS we just have to set options lnet lnet_peer_discovery_disabled=1 in order for it to mount all the lustre FS properly. Can you update 2.10.7 to 2.12.4 while jobs are running and FS are mounted and then reboot to get the new version, or do you have to remove the older version completely and install the new one with no lustre mounts ?</p> <p>Any other suggestions ?<br/> Thanks,<br/> Mike</p> <p>This is what I see on the OSS nodes:<br/> <span class="error">&#91;root@salt ~&#93;</span># salt 'holyscratch01oss*' cmd.run 'lctl get_param obdfilter.*.brw_size'<br/> holyscratch01oss05.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0000.brw_size=4<br/> obdfilter.scratch1-OST0006.brw_size=4<br/> obdfilter.scratch1-OST000c.brw_size=4<br/> obdfilter.scratch1-OST0012.brw_size=4<br/> obdfilter.scratch1-OST0018.brw_size=4<br/> obdfilter.scratch1-OST001e.brw_size=4<br/> obdfilter.scratch1-OST0024.brw_size=4<br/> obdfilter.scratch1-OST002a.brw_size=4<br/> holyscratch01oss02.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0004.brw_size=4<br/> obdfilter.scratch1-OST000a.brw_size=4<br/> obdfilter.scratch1-OST0011.brw_size=4<br/> obdfilter.scratch1-OST0016.brw_size=4<br/> obdfilter.scratch1-OST001d.brw_size=4<br/> obdfilter.scratch1-OST0023.brw_size=4<br/> obdfilter.scratch1-OST0029.brw_size=4<br/> obdfilter.scratch1-OST002f.brw_size=4<br/> holyscratch01oss06.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0001.brw_size=4<br/> obdfilter.scratch1-OST0007.brw_size=4<br/> obdfilter.scratch1-OST000d.brw_size=4<br/> obdfilter.scratch1-OST0013.brw_size=4<br/> obdfilter.scratch1-OST0019.brw_size=4<br/> obdfilter.scratch1-OST001f.brw_size=4<br/> obdfilter.scratch1-OST0025.brw_size=4<br/> obdfilter.scratch1-OST002b.brw_size=4<br/> holyscratch01oss04.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0003.brw_size=4<br/> obdfilter.scratch1-OST0009.brw_size=4<br/> obdfilter.scratch1-OST000f.brw_size=4<br/> obdfilter.scratch1-OST0015.brw_size=4<br/> obdfilter.scratch1-OST001b.brw_size=4<br/> obdfilter.scratch1-OST0021.brw_size=4<br/> obdfilter.scratch1-OST0027.brw_size=4<br/> obdfilter.scratch1-OST002d.brw_size=4<br/> holyscratch01oss01.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0005.brw_size=4<br/> obdfilter.scratch1-OST000b.brw_size=4<br/> obdfilter.scratch1-OST0010.brw_size=4<br/> obdfilter.scratch1-OST0017.brw_size=4<br/> obdfilter.scratch1-OST001c.brw_size=4<br/> obdfilter.scratch1-OST0022.brw_size=4<br/> obdfilter.scratch1-OST0028.brw_size=4<br/> obdfilter.scratch1-OST002e.brw_size=4<br/> holyscratch01oss03.rc.fas.harvard.edu:<br/> obdfilter.scratch1-OST0002.brw_size=4<br/> obdfilter.scratch1-OST0008.brw_size=4<br/> obdfilter.scratch1-OST000e.brw_size=4<br/> obdfilter.scratch1-OST0014.brw_size=4<br/> obdfilter.scratch1-OST001a.brw_size=4<br/> obdfilter.scratch1-OST0020.brw_size=4<br/> obdfilter.scratch1-OST0026.brw_size=4<br/> obdfilter.scratch1-OST002c.brw_size=4</p> <p>Note that "<tt>osc.&#42;.max_pages_per_rpc=1024</tt>" is 4MB (with <tt>PAGE_SIZE=4096</tt> on x86 clients). This matches with "<tt>obdfilter.&#42;.brw_size=4</tt>" on the OSTs. If you set "<tt>...max_pages_per_rpc=1M</tt>" it is internally converted to 256x 4KB pages. </p> <p>Hi,<br/> So nothing has worked so far so solve our problem with the bulk errors and we can only run on 4 of the 6 oss for now. We are going to try to update the lustre client side to 2.12.4 as soon as we can. Also the scratch01 lustre server that has issues is running 2.12.3 and at some point we will update that to 2.12.4.<br/> Thanks,<br/> Mike</p> <p>Hello,</p> <p>We upgraded all our 2150 compute node's lustre client to 2.12.4 this past Monday and it seems to have greatly stabilized things for us. Apparently this was the main fix we needed. So we have most devices (not all) on 2.12.4 (clients, lnet routers and Lustre FS). We have some main lustre FS on v2.12.3.</p> <p>We are planning on upgrading our compute nodes to the Centos 7.8 OS in October. Do you recommend we stay on 2.12.4 or update the lustre clients to 2.12.5 ? Our lustre storage will most likely stay on 2.12.3 or 2.12.4 for a while more.</p> <p>Thanks,<br/> Mike</p> <p>Mike</p> <p>I think that you'll need to move to 2.12.5 in order to get support for CentOS 7.8</p> <p>Peter</p> Development Rank 1|i011yv: Rank (Obsolete) 9223372036854775807 Severity