Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-9094] OOM caused by huge number of peers in case of INVALID_SERVICE_ID https://jira.whamcloud.com/browse/LU-9094 Lustre LU-9094

OOM caused by huge number of peers in case of INVALID_SERVICE_ID

Bug Major Resolved Fixed Doug Oucharek Sergey Cheremencev patch Thu, 9 Feb 2017 15:13:10 +0000 Wed, 1 Mar 2017 13:02:24 +0000 Wed, 1 Mar 2017 06:41:11 +0000 Lustre 2.10.0 0 5 Please change the Topic to something like "OOM caused by huge number of peers in case of INVALID_SERVICE_ID". Peer shouldn't be killed each time in case of INVALID_SERVICE_ID. This produces huge number of peers for the same nid and may cause an OOM. Issue could be simple reproduced using lctl ping to the node where lnet is not loaded(ib should be up). <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@pink03 tests]# cat ~/oom.sh while true; do lctl ping 172.18.56.129@o2ib0 done </pre> </div></div> The OOM was frequently seen with mlnx-ofa-kernel-2.3 where was used RCU mechanism in mlx4_cq_free. In older mlx4 versions to mitigate the issue mlx4_cq_free is reworked and doesn't use RCU anymore. Anyway we shouldn't create and remove tons of peers with the same nid to don't affect performance and memory. Also OOM issue should be reproducible on all mlx5 not depending on mlnx-ofa-kernel version. I reproduced it on mlnx-ofa_kernel-3.4 with mlx5. I prepared and tested a set of patches for it. Will send it in the nearest time. Sergey Cheremencev (sergey.cheremencev@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/25375" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25375</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> lnet: remove ni from lnet_finalize Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a312fcd635e63a79c53dd072ece9a066c1baf342 Sergey Cheremencev (sergey.cheremencev@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/25376" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25376</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> o2iblnd: kill timedout txs from ibp_tx_queue Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: fdee92bf4859793dc3fe4911b491ad9d0b21533e Sergey Cheremencev (sergey.cheremencev@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/25378" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25378</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> o2iblnd: reconnect peer for REJ_INVALID_SERVICE_ID Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 496c48a3daa21d0423a387625821de03a57db443 Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25378/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25378/</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> o2iblnd: reconnect peer for REJ_INVALID_SERVICE_ID Project: fs/lustre-release Branch: master Current Patch Set: Commit: 603aa7a1df6ee6ce6fe0d501a8b2bd1bfdf43bb8 Please take note that "reconnect peer for REJ_INVALID_SERVICE_ID" without "kill timedout txs from ibp_tx_queue" causes lctl ping hung when lnet is not loaded on the target node(lctl ping waits indefinitely). It was the reason why I pushed all 3 patches together. Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25375/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25375/</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> lnet: remove ni from lnet_finalize Project: fs/lustre-release Branch: master Current Patch Set: Commit: dab78a9efd05e4f22fc83232bdadce347d3dafda Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25376/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25376/</a> Subject: <a href="https://jira.whamcloud.com/browse/LU-9094" title="OOM caused by huge number of peers in case of INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-9094"><del>LU-9094</del></a> o2iblnd: kill timedout txs from ibp_tx_queue Project: fs/lustre-release Branch: master Current Patch Set: Commit: 824120da92fe8feb4b4308a136e33ec65fe3b635 Duplicate Development Rank 1|hzz3bj: Rank (Obsolete) 9223372036854775807 Severity