https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9368 Lustre <p>Scalability testing for 1/2/4/8/16 MDTs<br/> Works perfectly fine till 8 MDTs<br/> As moved to 16 MDTs saw following messages:<br/> Create snapshot for 16 MDTs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-52vm2 ~]# lctl snapshot_create -c TEST5 -F lustre -n MDT16 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host [root@eagle-52vm2 ~]# lctl snapshot_list -F lustre filesystem_name: lustre snapshot_name: MDT16 create_time: Thu Apr 20 05:28:21 2017 modify_time: Thu Apr 20 05:28:21 2017 comment: TEST5 snapshot_fsname: 5594a65 status: not mount filesystem_name: lustre snapshot_name: MDT8 modify_time: Thu Apr 20 05:14:13 2017 create_time: Thu Apr 20 05:14:13 2017 snapshot_fsname: 58754c5 comment: TEST4 status: not mount [root@eagle-52vm2 ~]# lctl snapshot_destroy -f -F lustre -n MDT8 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host [root@eagle-52vm2 ~]# lctl snapshot_list -F lustre filesystem_name: lustre snapshot_name: MDT16 create_time: Thu Apr 20 05:28:21 2017 modify_time: Thu Apr 20 05:28:21 2017 comment: TEST5 snapshot_fsname: 5594a65 status: not mount [root@eagle-52vm2 ~]# lctl snapshot_destroy -F lustre -n MDT16 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host Miss snapshot piece on the MDT0003. Use '-f' option if want to destroy it by force. Can't destroy the snapshot MDT16 [root@eagle-52vm2 ~]# lctl snapshot_destroy -f -F lustre -n MDT16 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host [root@eagle-52vm2 ~]# lctl snapshot_list -F lustre </pre> </div></div> <p>Even though the snapshot is created on creation and destroyed whenever the appropriate command is used, but still get the connection closed by remote host message. Only seen for 16 MDTs configuration, not for 1/2/4/8 MDTs.</p> 1 Client, 1 OST, 1/2/4/8/16 MDTs<br/> master, build# 3550 LU-9368

snapshot testing: ssh_exchange_identification: Connection closed by remote host

Bug Minor Closed Cannot Reproduce nasf Saurabh Tandan Thu, 20 Apr 2017 05:49:05 +0000 Mon, 4 Dec 2017 16:37:46 +0000 Mon, 4 Dec 2017 16:37:46 +0000 Lustre 2.10.0 0 3 <p>Hi Fan Yong,</p> <p>Can you please look into this snapshot issue?</p> <p>Thanks.<br/> Joe</p> <p>What is the output "lctl snapshot_list -F lustre -n MDT16 --detail"?</p> <p>Would you please to check whether you can "ssh" from current node to all other Lustre servers without password? Thanks!</p> <p>Yes I can ssh from MDS to OSS without password. Is vice-versa also required? Because I have not set up password less ssh from OSS to MDS.<br/> Earlier I had destroyed the snapshot for MDT16, so in order to give you details of "lctl snapshot_list -F lustre -n MDT16 --detail" i will have to re-create the snapshot. But when I am are-creating it I get following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-52vm2 ~]# lctl snapshot_create -c TEST5 -F lustre -n MDT16 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host cannot create snapshot 'lustre-mdt2/mdt2@MDT16': dataset already exists cannot create snapshot 'lustre-mdt11/mdt11@MDT16': dataset already exists ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host could not find any snapshots to destroy; check snapshot names. could not find any snapshots to destroy; check snapshot names. Can't create the snapshot MDT16 [root@eagle-52vm2 ~]# lctl snapshot_list -F lustre -n MDT16 --detail cannot open 'lustre-mdt1/mdt1@MDT16': dataset does not exist Can't list the snapshot MDT16 </pre> </div></div> <p>It only requires the auto ssh from current node to all Lustre servers (MGS/MDS/OSS).<br/> Would you please to use another name for another try, such as DMT16_2?</p> <p>Tried with another name. Got same result:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-52vm2 ~]# lctl snapshot_create -c TEST5 -F lustre -n MDT16_2 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host could not find any snapshots to destroy; check snapshot names. Can't create the snapshot MDT16_2 [root@eagle-52vm2 ~]# lctl snapshot_list -F lustre -n MDT16_2 --detail cannot open 'lustre-mdt1/mdt1@MDT16_2': dataset does not exist Can't list the snapshot MDT16_2 </pre> </div></div> <p>edit /etc/ssh/sshd_config, enable MaxStartups and set it as a larger value, such 128, then restart "sshd"</p> <p>Did as above but still got the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-52vm2 ~]# lctl snapshot_create -c TEST5 -F lustre -n MDT16 ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host ssh_exchange_identification: Connection closed by remote host Can't create the snapshot MDT16 [root@eagle-52vm2 ~]# </pre> </div></div> <p>It seems related with the 'ssh' configuration. Would you please to verify how many pure ssh connection (without lsnapshot) you can establish with the Lustre server nodes in parallel? Thanks!</p> <p>Have you restarted the sshd service after changed the "MaxStartups" ?<br/> Would you please to verify how many pure ssh connection (without lsnapshot) you can establish with the Lustre server nodes in parallel?</p> <p>Thanks!</p> <p>Saurabh,</p> <p>Any feedback for this? Thanks!</p> Development Rank 1|hzzanz: Rank (Obsolete) 9223372036854775807 Severity