https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14536 Lustre <p>when connecting to a host which is not up, for each discovery we will try retry_count(see kiblnd_check_reconnect) * lnet_retry_count(for resend) times.</p> <p>and for each ost when mounting the mdt, we will process attach add_conn(if ost has failover node) and add_osc so 3 times discovery.</p> <p>mounting of mdt when other nodes are not up can take very long, making customer think the mount is stuck.</p> LU-14536

kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID

Bug Minor Resolved Fixed Dongyang Li Dongyang Li Fri, 19 Mar 2021 09:50:13 +0000 Tue, 5 Apr 2022 03:15:29 +0000 Thu, 15 Apr 2021 17:31:45 +0000 Lustre 2.12.9 Lustre 2.15.0 0 7 <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/42109" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42109</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> o2iblnd: don't resend if there's no listener<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 29f79fe50652ac048949df4b8e4a2eafa235ebbf</p> <p>even without resend we are still retrying 5 times for each discovery, and for each ost from the conf llog we will try discovery once.</p> <p>I'm wondering should we even retry at all if there's no listener.</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/42111" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42111</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> obi2lnd: don't try to reconnect if there's no listener<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e164c7533c393be60ab35472743868eec6452129</p> <p>I manged to get access to the site experiencing the issue and got some numbers:<br/> Note the site has 415 OSTs, tested with latest 2.12</p> <p>when all the servers are up, mounting the targets on mds1:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lmds-vm1 o2iblnd]# cat mount.sh modprobe lnet modprobe lustre modprobe libcfs modprobe ksocklnd modprobe obdclass modprobe ptlrpc modprobe ldiskfs modprobe osd_ldiskfs modprobe ko2iblnd vgchange -ay vg_mdt0000_lustrefs --config <span class="code-quote">'activation{volume_list=[<span class="code-quote">"vg_mdt0000_lustrefs"</span>]}'</span> vgchange -ay vg_mgs --config <span class="code-quote">'activation{volume_list=[<span class="code-quote">"vg_mgs"</span>]}'</span> mount -t lustre -o max_sectors_kb=0 /dev/mapper/vg_mgs-mgs /lustre/mgs mount -t lustre -o max_sectors_kb=0 /dev/mapper/vg_mdt0000_lustrefs-mdt0000 /lustre/lustrefs/mdt0000 mount -t lustre -o max_sectors_kb=0 /dev/ddn/lustrefs_ost0000 /lustre/lustrefs/ost0000 mount -t lustre -o max_sectors_kb=0 /dev/ddn/lustrefs_ost0001 /lustre/lustrefs/ost0001 mount -t lustre -o max_sectors_kb=0 /dev/ddn/lustrefs_ost0400 /lustre/lustrefs/ost0400 mount -t lustre -o max_sectors_kb=0 /dev/ddn/lustrefs_ost0401 /lustre/lustrefs/ost0401 </pre> </div></div> <p>mounting mdt0000 took about 30mins</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [12863.422884] LNet: HW NUMA nodes: 1, HW CPU cores: 20, npartitions: 10 [12863.424484] alg: No test <span class="code-keyword">for</span> adler32 (adler32-zlib) [12864.250078] Lustre: Lustre: Build Version: 2.12.6 [12864.356817] LNet: Using FastReg <span class="code-keyword">for</span> registration [12864.396832] LNet: Added LNI 10.149.10.21@o2ib [8/640/0/180] [12864.434556] LNet: Added LNI 10.149.11.21@o2ib [8/640/0/180] [12864.895181] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelall oc [12866.435582] Lustre: MGS: Connection restored to 8529a39c-6bcb-c902-92ad-9af110ac39df (at 0@lo) [12866.819906] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,no_mbcache,node lalloc [14652.389489] Lustre: lustrefs-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [14652.395640] Lustre: lustrefs-MDT0000: in recovery but waiting <span class="code-keyword">for</span> the first client to connect [14675.527074] Lustre: lustrefs-MDT0000: Connection restored to 10.149.10.21@o2ib (at 0@lo) </pre> </div></div> <p>don't know how much time would it take for the OSTs on the host, mount script was terminated</p> <p>when it was working on mdt0000</p> <p>with patch 42109:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lmds-vm1 o2iblnd]# time mount.sh real 0m15.763s user 0m0.483s sys 0m6.796s </pre> </div></div> <p>with patch 42109 + 42111:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lmds-vm1 o2iblnd]# time mount.sh real 0m8.166s user 0m0.453s sys 0m6.703s</pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/42109/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42109/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> o2iblnd: don't resend if there's no listener<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0ab06eb9d865a47ea3e09880a41a9e8f0a78b6a6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/42111/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42111/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> obi2lnd: don't try to reconnect if there's no listener<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 67ba3ce23d32266eabd5f8c56fa78d65920455e8</p> <p>Landed for 2.15</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45510" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45510</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> o2iblnd: don't resend if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: 1<br/> Commit: 92acd551b2c97e2800181541e501815219bfc753</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45511" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45511</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> obi2lnd: don't try to reconnect if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: 1<br/> Commit: 33fe975fa96d9dbacda1e14c2147dfb276da12dd</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45895" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45895</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> o2iblnd: don't resend if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 9fe25415e52a73bde7e53871403266a5b5db859a</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45896</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> obi2lnd: don't try to reconnect if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 6ec51a85d49231e50694f0503406e548efbd6f17</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45895/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45895/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> o2iblnd: don't resend if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: da6e6953305be165798772d820ec59a0a209b604</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45896/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45896/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14536" title="kiblnd does resend for IB_CM_REJ_INVALID_SERVICE_ID" class="issue-link" data-issue-key="LU-14536"><del>LU-14536</del></a> obi2lnd: don't try to reconnect if there's no listener<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 3bd965a786afb04317acd1d8eb1708e594a1fc91</p> <p>The b2_14 backport causes servers to panic with null dereference error at MDT mount. Can we get this looked into please</p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/43069/43069_lustre.png" style="border: 0px solid black" /></span></p> Related Development Rank 1|i01py7: Rank (Obsolete) 9223372036854775807 Severity