https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15539 Lustre <p>We upgraded a lustre server cluster from lustre-2.12.7_2.llnl to lustre-2.12.8_6.llnl. </p> <p>The node on which the MGS runs, copper1, began reporting "new MDS connections" from NIDs that are assigned to client nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: MGS: Received new MDS connection from 192.168.128.68@o2ib38, keep former export from same NID Lustre: MGS: Received new MDS connection from 192.168.128.8@o2ib42, keep former export from same NID Lustre: MGS: Received new MDS connection from 192.168.131.78@o2ib39, keep former export from same NID Lustre: MGS: Received new MDS connection from 192.168.132.204@o2ib39, keep former export from same NID Lustre: MGS: Received new MDS connection from 192.168.134.127@o2ib27, keep former export from same NID </pre> </div></div> <p>Clients connect flags includes "mds_mds_connection":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@quartz7:lustre]# head */*/connect_flags ==&gt; mgc/MGC172.19.3.1@o2ib600/connect_flags &lt;== flags=0x2000011005002020 flags2=0x0 version barrier adaptive_timeouts mds_mds_connection full20 imp_recov bulk_mbits </pre> </div></div> <p>The clients are running lustre lustre-2.12.7_2.llnl, which does not have "<a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a> client: don't use OBD_CONNECT_MNE_SWAB".</p> <p>Shutting down the servers and restoring them to lustre-2.12.7_2.llnl did not change the symptoms.</p> <p>Patch stacks are:<br/> <a href="https://github.com/LLNL/lustre/releases/tag/2.12.8_6.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases/tag/2.12.8_6.llnl</a><br/> <a href="https://github.com/LLNL/lustre/releases/tag/2.12.7_2.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases/tag/2.12.7_2.llnl</a></p> <p>Seen during the same lustre server update where we saw <a href="https://jira.whamcloud.com/browse/LU-15541" title="Soft lockups in LNetPrimaryNID() and lnet_discover_peer_locked()" class="issue-link" data-issue-key="LU-15541">LU-15541</a> but appears to be a separate issue</p> lustre-2.12.8_6.llnl <br/> 3.10.0-1160.53.1.1chaos.ch6.x86_64 <br/> RHEL7.9 <br/> zfs-0.7.11-9.8llnl LU-15539

clients report mds_mds_connection in connect_flags after lustre update on servers

Bug Minor Resolved Fixed Lai Siyao Olaf Faaland llnl Wed, 9 Feb 2022 02:08:19 +0000 Fri, 1 Apr 2022 22:50:48 +0000 Mon, 21 Mar 2022 20:05:38 +0000 Lustre 2.12.8 0 6 <p>For my records, my local ticket is TOSS5543</p> <p>When we attempted to revert to lustre-2.12.7_2.llnl after observing the issue, we did <em>not</em> restore any server-side data; we just powered the servers down, booted them into the image with lustre-2.12.7_2.llnl, imported pools, and mounted the Lustre targets.</p> <p>After doing this, we continued to see the same symptoms.</p> <p>The clients which were still mounted (about 4,000) still had the incorrect connect flag from the earlier connection. This was <em>not</em> reset when they reconnected (or attempted to reconnect). I believe this connect flag not being reset may be why the symptoms did not change after the reboot.</p> <p>Perhaps, does <a href="https://jira.whamcloud.com/browse/LU-15453" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15453</a> need to be rolled out to the clients before it is rolled out to the servers?</p> <p>I am going to force-umount all the clients, before I try bringing the file system up on lustre-2.12.7_2.llnl, because I don't see another way to force the connect flags on the clients to change.</p> <p>I don't see evidence that this flag is recorded in the config logs somehow, and so I believe that a force umount on the client and then a mount of the file system running lustre-2.12.7_2.llnl will succeed in removing the mds_mds_connection flag. But if I'm wrong about that, please let me know ASAP.</p> <p>Thanks</p> <p>Most of these clients mount one or more other Lustre file systems (whose servers are running lustre-2.12.7_2.llnl and have not yet been upgraded to 2.12.8_6.llnl). On the example node I'm looking at, "lustre1" is the file system we tried to update, and "lustre2" is the other one which stayed at 2.12.7_2.llnl the whole time.</p> <p>On this client node, the connect_flags for the other file system, "lustre2", <em>also</em> shows "mds_mds_connection". </p> <p>Will I need to umount lustre2 and remount it to get the client to forget that connection flag?</p> <p>(There are two separate MGSs - one in the cluster that hosts "lustre1", and one in the cluster that hosts "lustre2", and each MGS knows only about the file system in the cluster it lives in. So I didn't expect this.)</p> <p>Olaf,</p> <p>Yes, I believe that should work.  (Re: the unmount plan)</p> <p>Thanks, Patrick.</p> <p>Patrick, I'd like to clarify which actions I've mentioned are still uncertain/need to be looked into, and which I should proceed with. So please confirm:</p> <p>(1) I should "umount -f /p/lustre1" (the file system that was briefly at 2.12.8_6.llnl) from every client that has it mounted?<br/> (2) Do I need to umount /p/lustre2 at the same time? <br/> (3) Since /p/lustre2 has been up the whole time, has jobs running against it, and was always at the "good" version, is there a less disruptive way you can think of?</p> <p>thanks</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Perhaps, does https://jira.whamcloud.com/browse/LU-15453 need to be rolled out to the clients before it is rolled out to the servers? </pre> </div></div> <p>Do you mean <a href="https://review.whamcloud.com/37880/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37880/</a> from <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a>? If so, yes, it should be rolled out to clients.</p> <p>Thank you, Lai</p> <p>Olaf,</p> <p>I would think (1) should be sufficient - the issue is an overlapping connect flag so servers without the new one should be unaffected.</p> <p>OK, thanks Patrick</p> <p>The mds_mds_connection flag is appearing in connect_flags on a client running lustre-2.12.7_2.llnl, which does not have patch 37880, on which I've umounted all lustre file systems, stopped lnet, verified that that libcfs is not loaded (nor any other dependent modules), started lnet, and mounted lustre2 whose servers are also all running lustre-2.12.7_2.llnl.</p> <p>This is incorrect, right?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zinci:~]# pdsh -a rpm -q lustre | dshbak -c ---------------- ezinc[1-52] ---------------- lustre-2.12.7_2.llnl-2.ch6.x86_64 [root@zinci:~]# pdsh -w e1 lctl list_nids e1: 172.19.3.1@o2ib600 [root@zinci:~]# pdsh -w e1 df -h -t lustre e1: Filesystem Size Used Avail Use% Mounted on e1: zinc1/mgs 2.7T 21M 2.7T 1% /mnt/lustre/MGS e1: zinc1/mdt1 2.8T 183G 2.7T 7% /mnt/lustre/lsh-MDT0000 </pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@quartz187:~]# pdsh -w e10 umount -a -t lustre [root@quartz187:~]# pdsh -w e10 systemctl stop lnet [root@quartz187:~]# pdsh -w e10 lsmod | grep libcfs [root@quartz187:~]# pdsh -w e10 systemctl start lnet [root@quartz187:~]# pdsh -w e10 mount -T /etc/fstab.d/ /p/lustre2 [root@quartz187:~]# pdsh -w e10 cat /sys/kernel/debug/lustre/mgc/MGC172.19.3.1@o2ib600/connect_flags e10: flags=0x2000011005002020 e10: flags2=0x0 e10: version e10: barrier e10: adaptive_timeouts e10: mds_mds_connection e10: full20 e10: imp_recov e10: bulk_mbits </pre> </div></div> <p>It looks correct to see this in 2.12.7, though it means MNE_SWAB here. After you land 37880, it should be gone.</p> <p>Lai, it seems as if the soft lockups in the description above are an LNet issue unrelated to the flag.  Do you agree?  If so, I'll create a separate Jira issue for it.</p> <p>Yes, it looks so.</p> <p>We umounted all the clients while the server cluster was down, then brought the server cluster back up in 2.12.7_2.llnl.  We did not see the "Received new MDS connection" messages on bringup.</p> <p>We are proceeding with client cluster updates, and will update server clusters in about 2 weeks.</p> <p>In retrospect the isssue that brought kept the lustre file system from coming up was the LNet issue documented in <a href="https://jira.whamcloud.com/browse/LU-15541" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15541</a> so reduced this issue priority to "Minor".</p> <p>Updated all the clients to 2.12.8_6.llnl (or later) with patch "<a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a> client: don't use OBD_CONNECT_MNE_SWAB".<br/> Then updated the servers to 2.12.8_6.llnl (or later) with that patch.<br/> No longer seeing inappropriate "Received new MDS connection" messages on bringup.</p> Related LU-13356 Development Rank 1|i02hvr: Rank (Obsolete) 9223372036854775807 Severity