https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6808 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/ff285e72-24dd-11e5-bf7b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ff285e72-24dd-11e5-bf7b-5254006e85c2</a>.</p> <p>The sub-test test_224c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>OST log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16:17:08:Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 16:16:38 (1436199398) 16:17:08:LustreError: 8997:0:(ldlm_lib.c:2718:target_bulk_io()) @@@ truncated bulk GET 1048576(4194304) req@ffff88003db95000 x1505956611442792/t0(0) o4-&gt;8e3c8004-e713-9f15-13d2-22232628f874@10.1.4.159@tcp:0/0 lens 608/448 e 0 to 0 dl 1436199422 ref 1 fl Interpret:/0/0 rc 0/0 16:17:08:Lustre: lustre-OST0004: Bulk IO write error with 8e3c8004-e713-9f15-13d2-22232628f874 (at 10.1.4.159@tcp), client will retry: rc -110 16:17:08:Lustre: lustre-OST0004: Client 8e3c8004-e713-9f15-13d2-22232628f874 (at 10.1.4.159@tcp) reconnecting 16:17:08:LustreError: 8990:0:(ldlm_lib.c:2718:target_bulk_io()) @@@ truncated bulk GET 1048576(3805696) req@ffff88004901f800 x1505956611442980/t0(0) o4-&gt;8e3c8004-e713-9f15-13d2-22232628f874@10.1.4.159@tcp:0/0 lens 608/448 e 0 to 0 dl 1436199427 ref 1 fl Interpret:/2/0 rc 0/0 16:17:08:LustreError: 8990:0:(ldlm_lib.c:2718:target_bulk_io()) Skipped 1 previous similar message 16:17:08:Lustre: lustre-OST0004: Bulk IO write error with 8e3c8004-e713-9f15-13d2-22232628f874 (at 10.1.4.159@tcp), client will retry: rc -110 16:17:08:Lustre: Skipped 1 previous similar message 16:21:30:LustreError: 8990:0:(ldlm_lib.c:2718:target_bulk_io()) @@@ truncated bulk GET 1048576(3805696) req@ffff88003596d000 x1505956611443196/t0(0) o4-&gt;8e3c8004-e713-9f15-13d2-22232628f874@10.1.4.159@tcp:0/0 lens 608/448 e 0 to 0 dl 1436199432 ref 1 fl Interpret:/2/0 rc 0/0 16:21:30:LustreError: 8990:0:(ldlm_lib.c:2718:target_bulk_io()) Skipped 1 previous similar message 16:21:30:Lustre: lustre-OST0004: Bulk IO write error with 8e3c8004-e713-9f15-13d2-22232628f874 (at 10.1.4.159@tcp), client will retry: rc -110 16:21:30:Lustre: Skipped 1 previous similar message </pre> </div></div> server: 2.5.3<br/> client: lustre-master build # 3092 EL7 LU-6808

Interop 2.5.3<->master sanity test_224c: Bulk IO write error

Bug Blocker Resolved Fixed Jinshan Xiong Maloo Tue, 7 Jul 2015 20:42:48 +0000 Thu, 1 Feb 2018 16:27:19 +0000 Tue, 25 Oct 2016 04:03:21 +0000 Lustre 2.8.0 Lustre 2.9.0 0 18 <p><a href="https://testing.hpdd.intel.com/test_sets/588f0ce8-6b5d-11e5-94a7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/588f0ce8-6b5d-11e5-94a7-5254006e85c2</a><br/> server:2.5.5<br/> client: 2.7.60/ build# 3203</p> <p>Instance also found for 2.7.63 tag with following config:<br/> Server: 2.7.0, b2_7/29<br/> Client: Master, build# 3251, RHEL 7<br/> <a href="https://testing.hpdd.intel.com/test_sets/fd0cce2c-90a1-11e5-a833-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fd0cce2c-90a1-11e5-a833-5254006e85c2</a></p> <p>Another instance for following interop config<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: Master, Build# 3266, Tag 2.7.64 <br/> <a href="https://testing.hpdd.intel.com/test_sets/8a0f2bb4-a04a-11e5-a33d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8a0f2bb4-a04a-11e5-a33d-5254006e85c2</a></p> <p>Server: 2.5.5, b2_5_fe/62<br/> Client: Master, Build# 3266, Tag 2.7.64 , RHEL 7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/fd8752d8-a05e-11e5-90cc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fd8752d8-a05e-11e5-90cc-5254006e85c2</a></p> <p>Another instance found for :<br/> Server: b2_7_fe/34<br/> Client: Master , Build# 3276<br/> <a href="https://testing.hpdd.intel.com/test_sets/79afc16e-a5e2-11e5-a028-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/79afc16e-a5e2-11e5-a028-5254006e85c2</a></p> <p>Another instance found for the following config:<br/> Server: 2.7.1 , b2_7_fe/34<br/> Client: Master, build# 3276, RHEL 6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/610eff92-a602-11e5-a14c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/610eff92-a602-11e5-a14c-5254006e85c2</a></p> <p>Another instance found for interop : 2.7.1 Server/EL7 Client<br/> Server: 2.7.1, b2_7_fe/34<br/> Client: master, build# 3303, RHEL 7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/23af3494-bb03-11e5-9137-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/23af3494-bb03-11e5-9137-5254006e85c2</a></p> <p>Another instance found for interop : 2.7.1 Server/EL6.7 Client<br/> Server: 2.7.1, b2_7_fe/34<br/> Client: master, build# 3303, RHEL 6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/04afed48-bb05-11e5-9137-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/04afed48-bb05-11e5-9137-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.7.1 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/5e49e47c-ccde-11e5-8b0e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5e49e47c-ccde-11e5-8b0e-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.7.1 Server/EL6.7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/c23f2696-ccdd-11e5-b80c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c23f2696-ccdd-11e5-b80c-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 -2.5.5 Server/EL6.7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/fad4bcda-ccf8-11e5-b1fa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fad4bcda-ccf8-11e5-b1fa-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.5.5 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/f385c93c-ccc7-11e5-b80c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f385c93c-ccc7-11e5-b80c-5254006e85c2</a></p> <p>Another instance found for interop - 2.7.1 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2</a><br/> Another instance found for interop - 2.7.1 Server/EL6.7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/f371534e-d573-11e5-bc47-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f371534e-d573-11e5-bc47-5254006e85c2</a><br/> Another instance found for interop - 2.5.5 Server/EL6.7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/e16b8c0c-d634-11e5-82a0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e16b8c0c-d634-11e5-82a0-5254006e85c2</a><br/> Another instance found for interop - 2.5.5 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2</a></p> <p>This issue has been seen almost 30 times in past 30 days</p> <p>Hi Emoly,<br/> Can you investigate this one?<br/> Thanks.<br/> Joe</p> <p>Is there any progress on this problem?</p> <p>We are trying to use 2.8.55 client (we need support for a 4.4 kernel) against our 2.5.41 servers and are hitting Bulk IO write errors (and Bulk IO read errors) very repeatedly on normal client activity.</p> <p>Can you reproduce this? If so can you run lctl set_param debug=+"net neterror nettrace" and collect debug logs.</p> <p>I can reproduce it easily (at least the IO read error)</p> <p>But i haven't used set_param debug before, so in case i need to do anything else you need to tell me exactly what.<br/> Meanwhile i'll do just the above on the client and the OSS:es where the problem shows up.</p> <p>Collected debug logs from both client and server.<br/> Where do you want them? client ~100M, server ~500M uncompressed</p> <p>Hi Ake,</p> <p>Please refer to <a href="https://wiki.hpdd.intel.com/display/WC/Uploading+Large+Logs" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/WC/Uploading+Large+Logs</a>. Thank you.</p> <p>Jian</p> <p>Ake will not be able to access that page on the wiki</p> <p>Ake</p> <p>It looks like you are using an Intel EE release rather than a community release so you should report this issue through your support channels.</p> <p>Peter</p> <p>The client is taken from git://git.hpdd.intel.com/fs/lustre-release.git using tag 2.8.55<br/> The server is from DDN.</p> <p>So, what route should i take?</p> <p>Hi Ake,</p> <p>Sorry for the late reply and thank you for collecting the debug logs.</p> <p>I reproduced the failure on the following combination and gathered full debug logs:<br/> Lustre client build: <a href="https://build.hpdd.intel.com/job/lustre-master/3425/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-master/3425/</a><br/> Lustre server build: <a href="https://build.hpdd.intel.com/job/lustre-b2_7_fe/95/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_7_fe/95/</a></p> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/8cc945de-6d71-11e6-8afd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8cc945de-6d71-11e6-8afd-5254006e85c2</a></p> <p>Hi Nasf,</p> <p>The server side patch <a href="http://review.whamcloud.com/14399" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14399</a> was included in build <a href="https://build.hpdd.intel.com/job/lustre-b2_7_fe/95/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_7_fe/95/</a>. Could you please look into the issue? Thank you.</p> <p>The root reason is the interoperability issue introduced by the patch <a href="http://review.whamcloud.com/15421" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15421</a>. In such patch, we replaced the "rq_xid" with "rq_mbits" as matchbits of bulk data. For be interoperable with old server, it introduced new connection flag: OBD_CONNECT_BULK_MBITS. If the server does not support such feature, then the "rq_xid" would be set the same value as "rq_mbits". Unfortunately, it forgot to handle multiple bulk operations, for example 4MB IO. If the new client (such as master, b2_8_fe) wants to make 4MB IO, then the new client may send a small "rq_xid" to the old server (such as b2_7_fe, IEEL2), as to the old server will regard it as an 1MB IO or 2MB IO. So the data transfer will NOT complete because of only part of data transferred. Then the client will get timeout failure and retry again and again.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22373" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22373</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> ptlrpc: properly set "rq_xid" for 4MB IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 56766651d8a124156c615aef60c850cf41d9d5e6</p> <p>That patch fixes our problem at least.<br/> I.e., I can't trigger it the way I could before so I'm very happy.<br/> Thanks alot.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22373/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22373/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> ptlrpc: properly set "rq_xid" for 4MB IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bb75072cb679bf52e00537c19e42f8e4e95255b6</p> <p>Landed for 2.9</p> <p>Reopen to get fix <a href="http://review.whamcloud.com/22906" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22906</a> landed for 2.9.0.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22957" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22957</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> ptlrpc: BULK_BITS recovery for old servers<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d58ad7550497e5766fe46582272ca3d2636bd0f5</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23048" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23048</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> ptlrpc: no need to reassign mbits for replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 487586d8d40506f1e9980ecd8f8aad5b5277f519</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23048/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23048/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> ptlrpc: no need to reassign mbits for replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 48488fa271e4444e4a2ab4f3babb6c91b779634e</p> <p>Landed for 2.9</p> <p>Just finished running into this in the in-tree lustre client driver for the 4.12 kernel.</p> <p>That is, larger reading/writing operations on a mount would hang any further file reading/writing operations on that mount and the system would endlessly spit</p> <p>...<br/> Connection to &lt;OST&gt; was lost; in progress operations using this service will wait for recovery to complete <br/> Connection restored to &lt;OST&gt; <br/> Connection to &lt;OST&gt; was lost; in progress operations using this service will wait for recovery to complete <br/> Connection restored to &lt;OST&gt; <br/> ...</p> <p>Applying the two merged patches from this ticket resolved the issue</p> <p><a href="https://review.whamcloud.com/#/c/22373/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/22373/</a><br/> <a href="https://review.whamcloud.com/#/c/23048/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/23048/</a></p> <p>It would be great if these could both see their way into the in-tree lustre client driver.</p> <p>Thanks! -Tyson</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> are these patches on your radar?</p> <p>No. I missed those. Will port them to upstream client.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=twhitehead" class="user-hover" rel="twhitehead">twhitehead</a> if you have already ported those patches to the upstream kernel, please feel free to submit the patches directly to the mailing list. This has the advantage that:</p> <ul class="alternate" type="square"> <li>James doesn't have to do more work to port and test the patches himself</li> <li>it shows the kernel maintainers that people are interested and using the upstream kernel code</li> <li>you become familiar with the process yourself and can potentially help with other patches in the future</li> </ul> <p>See <a href="https://wiki.hpdd.intel.com/display/PUB/Commit+Comments#Additional_commit_tags" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/PUB/Commit+Comments#Additional_commit_tags</a> for labels to include in patches ported from the Lustre master tree to the upstream kernel, and the script <tt>scripts/get_maintainer.pl drivers/staging/lustre</tt> to get a list of email addresses to sent patches to.</p> <p>Will do. The script gives me 7 addresses, 3 of which are lists. Do I send it to all of them?</p> <p>With regard to the commit message, I just did a format-patch on the two commit above and then applied them with am and fixed up the issues. This retained the entire original commit message. It references the issue number, but it doesn't include any info about the exact original commit.</p> <p>Is this sufficient, or should I be trying to add something more?</p> <p>Thanks! -Tyson</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=twhitehead" class="user-hover" rel="twhitehead">twhitehead</a>, thanks for following up on this. The <a href="https://wiki.hpdd.intel.com/display/PUB/Commit+Comments" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/PUB/Commit+Comments</a> page has a section <tt>Patch porting examples</tt> that lists the commit comment tags that should be used when porting a patch to the upstream kernel. The patch should be sent to at least Greg KH, CC Oleg, James, and myself, along with <tt>lustre-devel</tt> and <tt>devel</tt>.</p> <p>Ok. Have it ready to go, but am holding off because I would like to know everything is stable first and we are still experiencing instabilities that I believe are lustre related.</p> <p>That is, bulk operations seems okay now, but within 24hrs the machine falls off the network and the kernel spits soft lockup messages with osc_io_unplug0 showing up in the backtrace.</p> <p><a href="https://photos.app.goo.gl/zcbG6GGmB0eBgo0G3" class="external-link" target="_blank" rel="nofollow noopener">https://photos.app.goo.gl/zcbG6GGmB0eBgo0G3</a></p> <p>I believe it is possibly <a href="https://jira.whamcloud.com/browse/LU-8680" title="replay-single test_20b: BUG: soft lockup - osc_makes_rpc()" class="issue-link" data-issue-key="LU-8680"><del>LU-8680</del></a>, so I've ported that patch as well. Just rebooted it into the new kernel last night. So far it is still up with no signs of problems. Fingers crossed!</p> <p>I already push <a href="https://jira.whamcloud.com/browse/LU-8680" title="replay-single test_20b: BUG: soft lockup - osc_makes_rpc()" class="issue-link" data-issue-key="LU-8680"><del>LU-8680</del></a> but it wouldn't show up in Linus tree until the 4.14-rc1 merge window. Luckly 4.14 is a LTS kernel so the client will be usable for general users. The tree to work against is the staging tree. Specifically the staging-next branch. Normally the staging-tree is about 100+ patches ahead of Linus tree at any time. If you need help moving to the staging tree let me know. I will gladly help.</p> <p>Feel free to send me the patches for the upstream client. I will look at them. My email is in the MAINTAINERS file for lustre (James Simmons)</p> <p>Just a quick follow up on this to let you know I believe things are now stable in 4.12 with this and the LU 8680 patch.</p> <p>The machine has been up under load for 8 days now without issue. Prior to these patches it wouldn't last 24hrs. Very happy about this. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I've ported the patch to staging (and, IIRC, emailed it to James), but am not in a position to test as we really need to avoid any further downtime on this system.</p> <p>Thanks! -Tyson</p> <p>Sorry I have been busy with sysfs porting/fixing. So I looked at the latest staging I see one of the patches for <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> already landed. I missed the other one. I just got the latest staging-next tree to build and I'm looking to port stuff so I will include your other patch. Thanks.</p> <p>Rebuilt for the spectre/meltdown bugs and discovered this hasn't full landed in upstream yet. Specifically, if I try and do something like a copy on a larger file, it just hangs while the kernel endlessly spits messages like these</p> <p><span class="error">&#91;60679.700856&#93;</span> Lustre: covework-OST0017-osc-ffff888d70ebf800: Connection to covework-OST0017 (at 10.18.26.8@tcp) was lost; in progress operations using this service will wait for recovery to complete<br/> <span class="error">&#91;60679.719316&#93;</span> Lustre: Skipped 49 previous similar messages<br/> <span class="error">&#91;60679.740153&#93;</span> Lustre: covework-OST0017-osc-ffff888d70ebf800: Connection restored to covework-OST0017 (at 10.18.26.8@tcp)<br/> <span class="error">&#91;60679.751843&#93;</span> Lustre: Skipped 49 previous similar messages<br/> <span class="error">&#91;61279.724063&#93;</span> Lustre: 1144:0:(client.c:2068:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1517498129/real 1517498129&#93;</span> req@ffff888d3c128c00 x1591147922869136/t0(0) o3-&gt;covework-OST0017-osc-ffff888d70ebf800@10.18.26.8@tcp:6/4 lens 608/432 e 0 to 1 dl 1517498141 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> <span class="error">&#91;61279.756551&#93;</span> Lustre: 1144:0:(client.c:2068:ptlrpc_expire_one_request()) Skipped 49 previous similar messages</p> <p>Dug through the prior patches and compared against the current master kernel. Looks like part of the the patch set never landed upstream</p> <p><a href="https://review.whamcloud.com/#/c/22373/5/lustre/ptlrpc/client.c" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/22373/5/lustre/ptlrpc/client.c</a><br/> <a href="https://github.com/torvalds/linux/blob/255442c93843f52b6891b21d0b485bf2c97f93c3/drivers/staging/lustre/lustre/ptlrpc/client.c#L3133" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/torvalds/linux/blob/255442c93843f52b6891b21d0b485bf2c97f93c3/drivers/staging/lustre/lustre/ptlrpc/client.c#L3133</a><br/> <a href="https://github.com/torvalds/linux/blob/255442c93843f52b6891b21d0b485bf2c97f93c3/drivers/staging/lustre/lustre/ptlrpc/client.c#L3163" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/torvalds/linux/blob/255442c93843f52b6891b21d0b485bf2c97f93c3/drivers/staging/lustre/lustre/ptlrpc/client.c#L3163</a></p> <p>Thanks! -Tyson</p> Duplicate Related LU-6441 LU-484 Development Rank 1|hzxhm7: Rank (Obsolete) 9223372036854775807 Severity