https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8867 Lustre <p>From customer Log, We found LNetError reports as below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 13 17:06:58 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff8819375e0000) timed out 2 secs ago, resid: 0, wmem: 0 Nov 18 02:22:37 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 1 secs ago, resid: 0, wmem: 3096232 Nov 18 02:22:38 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 1 secs ago, resid: 0, wmem: 3096232 Nov 18 02:22:39 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 2 secs ago, resid: 0, wmem: 3089888 Nov 18 02:22:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 4 secs ago, resid: 0, wmem: 3090440 Nov 18 02:22:45 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 8 secs ago, resid: 0, wmem: 3090440 Nov 18 02:22:53 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 16 secs ago, resid: 0, wmem: 3090440 Nov 18 02:23:09 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 32 secs ago, resid: 0, wmem: 3090440 Nov 18 02:23:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 64 secs ago, resid: 0, wmem: 3090440 Nov 18 02:24:45 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 128 secs ago, resid: 0, wmem: 3090440 Nov 18 02:26:53 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 256 secs ago, resid: 0, wmem: 3090440 Nov 18 02:31:09 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 512 secs ago, resid: 0, wmem: 3090440 Nov 18 02:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 1024 secs ago, resid: 0, wmem: 3090440 Nov 18 02:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 1624 secs ago, resid: 0, wmem: 3090440 Nov 18 02:59:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 2224 secs ago, resid: 0, wmem: 3090440 Nov 18 03:09:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 2824 secs ago, resid: 0, wmem: 3090440 Nov 18 03:19:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 3424 secs ago, resid: 0, wmem: 3090440 Nov 18 03:29:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 4024 secs ago, resid: 0, wmem: 3090440 Nov 18 03:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 4624 secs ago, resid: 0, wmem: 3090440 Nov 18 03:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 5224 secs ago, resid: 0, wmem: 3090440 Nov 18 03:59:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 5824 secs ago, resid: 0, wmem: 3090440 Nov 18 04:09:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 6424 secs ago, resid: 0, wmem: 3090440 Nov 18 04:19:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 7024 secs ago, resid: 0, wmem: 3090440 Nov 18 04:29:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 7624 secs ago, resid: 0, wmem: 3090440 Nov 18 04:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 8224 secs ago, resid: 0, wmem: 3090440 Nov 18 04:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 8824 secs ago, resid: 0, wmem: 3090440 Nov 18 04:59:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 9424 secs ago, resid: 0, wmem: 3090440 Nov 18 05:09:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 10024 secs ago, resid: 0, wmem: 3090440 Nov 18 05:19:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 10624 secs ago, resid: 0, wmem: 3090440 Nov 18 05:29:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 11224 secs ago, resid: 0, wmem: 3090440 Nov 18 05:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 11824 secs ago, resid: 0, wmem: 3090440 Nov 18 05:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 12424 secs ago, resid: 0, wmem: 3090440 Nov 18 05:59:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 13024 secs ago, resid: 0, wmem: 3090440 Nov 18 06:09:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 13624 secs ago, resid: 0, wmem: 3090440 Nov 18 06:19:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 14224 secs ago, resid: 0, wmem: 3090440 Nov 18 06:29:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 14824 secs ago, resid: 0, wmem: 3090440 Nov 18 06:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 15424 secs ago, resid: 0, wmem: 3090440 Nov 18 06:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 16024 secs ago, resid: 0, wmem: 3090440 Nov 18 06:59:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 16624 secs ago, resid: 0, wmem: 3090440 Nov 18 07:09:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 17224 secs ago, resid: 0, wmem: 3090440 Nov 18 07:19:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 17824 secs ago, resid: 0, wmem: 3090440 Nov 18 07:29:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 18424 secs ago, resid: 0, wmem: 3090440 Nov 18 07:39:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 19024 secs ago, resid: 0, wmem: 3090440 Nov 18 07:49:41 OSS04 kernel: LNetError: 3721:0:(socklnd_cb.c:2519:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 192.235.3.34@tcp detected; the oldest(ffff881a2535a000) timed out 19624 secs ago, resid: 0, wmem: 3090440 </pre> </div></div> <p>It is abnormal case to close same TX multiple times. </p> LU-8867

Ignore timedout TX on closing connection

Bug Minor Resolved Fixed Yang Sheng Yang Sheng Mon, 28 Nov 2016 07:33:15 +0000 Thu, 22 Dec 2016 20:36:48 +0000 Sat, 17 Dec 2016 15:22:09 +0000 Lustre 2.10.0 0 2 <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23973" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23973</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8867" title="Ignore timedout TX on closing connection " class="issue-link" data-issue-key="LU-8867"><del>LU-8867</del></a> ksocklnd: ignore timedout TX on closing connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 22ffe085a98f00b9497506768ce71015b4c93dce</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23973/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23973/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8867" title="Ignore timedout TX on closing connection " class="issue-link" data-issue-key="LU-8867"><del>LU-8867</del></a> ksocklnd: ignore timedout TX on closing connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 21e6b6edbc95eef46c241335147818f1df871fa5</p> <p>Landed for 2.10</p> Related Development Rank 1|hzywjb: Rank (Obsolete) 9223372036854775807 Severity