Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-15915] /bin/rm: fts_read failed: Cannot send after transport endpoint shutdown https://jira.whamcloud.com/browse/LU-15915 Lustre <p>Am running a large number of deletes on clients and after a while they get evicted, the error on the client is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /bin/rm: fts_read failed: Cannot send after transport endpoint shutdown </pre> </div></div> <p>On the MDS, the error is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> un 6 19:28:59 fmds1 kernel: LustreError: 9744:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.21.22.31@tcp ns: mdt-foxtrot-MDT0000_UUID lock: ffff94f72a408480/0xb4442ee3e798319c lrc: 3/0,0 mode: PR/PR res: [0x20009b3c6:0x29eb:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.21.22.31@tcp remote: 0x40ff70b2e6a5419f expref: 147862 pid: 61992 timeout: 6578337 lvb_type: 0 </pre> </div></div> <p>I'm running maybe 10-15 recursive rm on 3 clients, so 30-45 in total at once. </p> <p>I've set debugging params as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param debug_mb=1024 lctl set_param debug="+dlmtrace +info +rpctrace" lctl set_param dump_on_eviction=1 </pre> </div></div> <p>on clients and the MDS.</p> <p>Lustre version is 2.12.8_6_g5457c37</p> LU-15915 /bin/rm: fts_read failed: Cannot send after transport endpoint shutdown Bug Minor Resolved Duplicate Lai Siyao Dneg Mon, 6 Jun 2022 19:23:05 +0000 Fri, 25 Nov 2022 13:48:50 +0000 Sat, 19 Nov 2022 16:22:53 +0000 Lustre 2.12.8 0 6 <p>should one of the clients get evicted again, I'll upload the logs</p> <p>Several evictions last night. I've uploaded the logs to ftp.whamcloud.com uploads/<a href="https://jira.whamcloud.com/browse/LU-15915" title="/bin/rm: fts_read failed: Cannot send after transport endpoint shutdown" class="issue-link" data-issue-key="LU-15915"><del>LU-15915</del></a>.</p> <p>Could we please raise the priority on this? Thanks</p> <p>Are you still experiencing this issue? <del>There isn't enough technical information to diagnose the problem.</del> (We just noticed your follow-on comment about the Lustre logs uploaded to ftp.whamcloud.com and found your logs still there).</p> <p>Can you please attach an hour or two of console logs preceding the client eviction from the MDS and from the affected clients? Also, the Lustre kernel debug logs if they are available.</p> <p>Based on the description of the problematic workload (concurrent "<tt>rm -r</tt>" from many clients) it is possible that this relates to <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a>.</p> <p>The patch <a href="https://review.whamcloud.com/47215" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47215</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks</tt>" prioritizes the processing of lock callbacks from the servers, which can be problematic if there is a workload that is continually creating new locks and preventing the lock callbacks from the server from being processed in a timely manner.</p> <p>Would you be interested to test this simple patch on the clients that are running the "<tt>rm -r</tt>" tasks? It is very low risk and unlikely to cause any problems, but is not totally sure whether it will solve the problem. </p> <p>Our current workaround is to set the lru_size to 128 (ldlm.namespaces.*.lru_size=128), on the principle that having less locks allow the client and MDS to communicate all the locks in a timely manner, which seems to have reduced the evictions (though it means a lot more getattr calls to the MDS), and things are slower. </p> <p>How would we apply this patch? Should I download srpms, apply the patch and build? If so, which srpms need rebuilding? Can I patch the same release (2.12.8_6_g5457c37) as we're currently using? And is it client only? Or for the servers as well? </p> <p>The patch is only on the client, and should be able to apply to the source you are currently using and then rebuild the client RPMs. My assumption, based on the build version in use on your clients, is that you already build your own client RPMs, so this should not be any different.</p> <p>Thanks Andreas. Actually, we use Whamcloud-supplied rpms, so I'll download the corresponding srpm(s)</p> <p>I've cherry-picked patch <a href="https://review.whamcloud.com/48764" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48764</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks</tt>" to b2_12 to generate a build and run some testing.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dneg" class="user-hover" rel="dneg">dneg</a> I'm not sure how familiar you are in navigating the Whamcloud development infrastructure but you should now be able to grab SRMs from Jenkins - <a href="https://build.whamcloud.com/job/lustre-reviews/89806/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/89806/</a></p> <p>Hi Peter, </p> <p>I found the rpms from that link (e.g., <a href="https://build.whamcloud.com/job/lustre-reviews/89806/arch=x86_64,build_type=client,distro=el7.9,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/kmod-lustre-client-2.12.9_11_gf855161-1.el7.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/89806/arch=x86_64,build_type=client,distro=el7.9,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/kmod-lustre-client-2.12.9_11_gf855161-1.el7.x86_64.rpm</a> etc), but unless you recommend using that later version (2.12.9) I was going to use the same version as we're running, which I've downloaded the source rpms for, and applied the patch Andreas linked then built the client rpms. Let me know if that is ok.</p> <p>Thanks,<br/> Campbell</p> <p>Yes that is fine - I was just trying to make things easier for you.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dneg" class="user-hover" rel="dneg">dneg</a> just checking in to see whether you have tried the suggested patch yet...</p> <p>Hi Peter, have just managed to apply the patched rpm to the clients now, and have increased the lru_size back up to 10000 from the 128 we had it set to, across the cluster. It took a long time as I had to wait for all the long running backups on these clients to finish, which I had to do one by one. So we should get some results through in the next couple of days, and I'll let you know either way.<br/> -Campbell</p> <p>Great - thanks!</p> <p>Got three evictions last night on one of the three patched clients:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Oct 20 01:54:22 foxtrot2 kernel: LustreError: 167-0: foxtrot-MDT0000-mdc-ffff99c032e95800: This client was evicted by foxtrot-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail. Oct 20 03:02:17 foxtrot2 kernel: LustreError: 167-0: foxtrot-MDT0000-mdc-ffff99c032e95800: This client was evicted by foxtrot-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail. Oct 20 03:38:09 foxtrot2 kernel: LustreError: 167-0: foxtrot-MDT0000-mdc-ffff99c032e95800: This client was evicted by foxtrot-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail. </pre> </div></div> <p>3 more evictions last night, two on one client, one on another. </p> <p>I just realized you did not include the server side logs so we don't know which logs the client was supposed to release and did not in time.</p> <p>Can you please provide the MDS logs from the timeframe when those lustre-log files you uploaded were created?</p> <p>Hi Oleg,</p> <p>I've uploaded the logs from the mds from the earlier June evictions, as well as the latest batch, to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-15915" title="/bin/rm: fts_read failed: Cannot send after transport endpoint shutdown" class="issue-link" data-issue-key="LU-15915"><del>LU-15915</del></a>. I've counted altogether 39 evictions in the 6 days since the patches were applied and lru_size was increased back to to 10000. I've dropped it back down to 2048 to see if that makes much of a difference, though it might increase the load on the mds due to increased numbers of getattr calls, guess we'll see.</p> <p>Campbell</p> <p>Dropped the lru_size to 128 as we were still getting evictions and backups were failing.</p> <p>so these (June) logs look very familiar, I think there was a similar report  some time ago where a lock cancel was triggering other lock cancels ? Can't quite find the report readily though to refresh my memory. ...  </p> <p>On the surface it sounds like <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> but I think there was another path where as we do a cancel we also want to pack up some more locks that are ready to be cancelled (I cannot verify what your current client version is to see it really got the fix and I don't have more recent debug logs to confirm the behavior is same or different now).</p> <p>In the old logs this is how it unfolds:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00010000:9.0:1654595833.299427:0:3716:0:(ldlm_lockd.c:1775:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: foxtrot-MDT0000-mdc-ffff8acff2c13800 lock: ffff8af441314d80/0x7960025dca1d30a8 lrc: 2/0,0 mode: PR/PR res: [0x20009b4d8:0x14677:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x400000000000 nid: local remote: 0xb4442ee9c835c045 expref: -99 pid: 35329 timeout: 0 lvb_type: 0 00010000:00010000:9.0:1654595833.299485:0:3716:0:(ldlm_request.c:1150:ldlm_cli_cancel_local()) ### client-side cancel ns: foxtrot-MDT0000-mdc-ffff8acff2c13800 lock: ffff8af441314d80/0x7960025dca1d30a8 lrc: 3/0,0 mode: PR/PR res: [0x20009b4d8:0x14677:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT flags: 0x408400000000 nid: local remote: 0xb4442ee9c835c045 expref: -99 pid: 35329 timeout: 0 lvb_type: 0 00010000:00000040:9.0:1654595833.299489:0:3716:0:(ldlm_resource.c:1601:ldlm_resource_putref()) putref res: ffff8b0921172e40 count: 2 00010000:00000040:24.0:1654595833.299492:0:35567:0:(ldlm_resource.c:1601:ldlm_resource_putref()) putref res: ffff8afaf44dac00 count: 1 00000020:00000040:9.0:1654595833.299493:0:3716:0:(lustre_handles.c:113:class_handle_unhash_nolock()) removing object ffff8af441314d80 with handle 0x7960025dca1d30a8 from hash ... lots and lots of other locks are being collected to be sent in the cancel RPC ... 00010000:00010000:4.0:1654595933.168282:0:3716:0:(ldlm_lockd.c:1800:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: foxtrot-MDT0000-mdc-ffff8acff2c13800 lock: ffff8af441314d80/0x7960025dca1d30a8 lrc: 1/0,0 mode: --/PR res: [0x20009b4d8:0x14677:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT flags: 0x4c09400000000 nid: local remote: 0xb4442ee9c835c045 expref: -99 pid: 35329 timeout: 0 lvb_type: 0</pre> </div></div> <p>So from this we see we already traversed the ldlm_cli_cancel_local-&gt;ldlm_lock_cancel-&gt;ldlm_lock_destroy_nolock-&gt;class_handle_unhash_nolock - AKA the lock was totally destroyed and it's not a matter of the cancel thread not even getting to it as in <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a></p> <p>anyway, looking closer at this thread (which is named ldlm_bl_126) we actually see this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00100000:11.0:1654595833.301719:0:3716:0:(genops.c:2379:obd_get_mod_rpc_slot()) foxtrot-MDT0000-mdc-ffff8acff2c13800: sleeping for a modify RPC slot opc 35, max 7 00000100:00100000:4.0:1654595933.167133:0:3716:0:(client.c:2096:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ldlm_bl_126:22bdcfdc-5f27-ca71-fcf9-840efc5add00:3716:1733599145210368:10.21.22.10@tcp:35 </pre> </div></div> <p>so somewhere along the way of canceling multiple locks it suddenly met one that required a modify rpc slot that we did not have. opc 35 is MDS_CLOSE, so we were canceling an open lock and could not send the close stopping this whole canceling thread in it's tracks.</p> <p>So this is actually <a href="https://jira.whamcloud.com/browse/LU-14741" title="Close RPC might get stuck behind normal RPCs waiting for slot" class="issue-link" data-issue-key="LU-14741"><del>LU-14741</del></a> that was never included into b2_12, the backported patch is here: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45850" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45850</a></p> <p>Hi Oleg, I have applied the patch to lustre/obdclass/genops.c (there was just the one at <a href="https://review.whamcloud.com/changes/fs%2Flustre-release~45850/revisions/1/patch?zip&amp;path=lustre%2Fobdclass%2Fgenops.c" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/changes/fs%2Flustre-release~45850/revisions/1/patch?zip&amp;path=lustre%2Fobdclass%2Fgenops.c</a>, correct?) and have built new client rpms. I'll install them on the clients over the next few days, then bump up the lru_size across the cluster and let you know the result.<br/> Thanks,<br/> Campbell</p> <p>Hey Campbell</p> <p>Just checking in to see how things are progressing</p> <p>Peter</p> <p>Hi Peter, was just about to post an update. No evictions since the patch was applied ealier in the week (Tuesday), so good news on that front. Will keep an eye on it over the weekend. We get the odd soft lockup (e.g., Nov 9 03:11:25 foxtrot3 kernel: NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! <span class="error">&#91;ptlrpcd_01_10:3531&#93;</span>). I can open a separate ticket for that issue if you like</p> <p>I saw <a href="https://jira.whamcloud.com/browse/LU-15742" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15742</a>, we already have lru_size at 10000, and ldlm.namespaces.*.lru_max_age=60000</p> <p>Looking good, still no evictions after a week. </p> <p>Great! Then let's mark this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-14741" title="Close RPC might get stuck behind normal RPCs waiting for slot" class="issue-link" data-issue-key="LU-14741"><del>LU-14741</del></a>. It would be better to track the soft lockup issue under a new ticket.</p> <p>Thanks Peter, can you tell me which releases (in particular, 2.15.x and 2.12.x) have this genops.c patch?</p> <p>This fix was in 2.15.0 and will be in 2.12.10 (if we do one)</p> Duplicate LU-14741 Related LU-14741 LU-15821 Development Rank 1|i02rhz: Rank (Obsolete) 9223372036854775807 Severity