https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5583 Lustre <p>After our active MDS became completely unresponsive earlier, we attempted to fail over to the second MDS. This appeared to succeed, the MGS and MDT mounted successfully, as far as we can tell all clients reconnected, recovery completed. However at this stage, any operation on the file system (for example ls) on any client only connected via ethernet either hung or returned I/O errors, all clients using IB were operating normally.</p> <p>We then discovered that the MDT that there seemed to be a problem between MDT and all OSTs, as <tt>lctl get_param lod.lustre03-MDT0000-mdtlov.target_obd</tt> came back empty. Failing back to the (now rebooted) previous MDT worked and the file system is now operating normally again.</p> <p>Sample errors in syslog on one of the ethernet only clients while <tt>ls /mnt/lustre03</tt> was returing I/O errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 4 09:56:18 cs04r-sc-serv-06 kernel: Lustre: MGC172.23.144.1@tcp: Connection restored to MGS (at 172.23.144.2@tcp) Sep 4 09:57:58 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 09:58:23 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 09:58:48 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 09:59:13 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 09:59:38 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:00:03 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:00:28 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:01:18 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:01:18 cs04r-sc-serv-06 kernel: LustreError: Skipped 1 previous similar message Sep 4 10:02:33 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:02:33 cs04r-sc-serv-06 kernel: LustreError: Skipped 2 previous similar messages Sep 4 10:05:03 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:05:03 cs04r-sc-serv-06 kernel: LustreError: Skipped 5 previous similar messages Sep 4 10:09:38 cs04r-sc-serv-06 kernel: LustreError: 11-0: lustre03-MDT0000-mdc-ffff880073fec800: Communicating with 172.23.144.2@tcp, operation mds_connect failed with -16. Sep 4 10:09:38 cs04r-sc-serv-06 kernel: LustreError: Skipped 10 previous similar messages Sep 4 10:33:15 cs04r-sc-serv-06 kernel: LustreError: 32662:0:(dir.c:422:ll_get_dir_page()) read cache page: [0xe900001:0x3b1189d1:0x0] at 0: rc -4 Sep 4 10:33:15 cs04r-sc-serv-06 kernel: LustreError: 32662:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -4 Sep 4 10:34:00 cs04r-sc-serv-06 kernel: LustreError: 32717:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:34:00 cs04r-sc-serv-06 kernel: LustreError: 32717:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:37:44 cs04r-sc-serv-06 kernel: LustreError: 487:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -4 Sep 4 10:37:44 cs04r-sc-serv-06 kernel: LustreError: 487:0:(mdc_locks.c:918:mdc_enqueue()) Skipped 879 previous similar messages Sep 4 10:37:57 cs04r-sc-serv-06 kernel: LustreError: 508:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:37:57 cs04r-sc-serv-06 kernel: LustreError: 508:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:37:58 cs04r-sc-serv-06 kernel: LustreError: 510:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:37:59 cs04r-sc-serv-06 kernel: LustreError: 512:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:37:59 cs04r-sc-serv-06 kernel: LustreError: 512:0:(dir.c:584:ll_dir_read()) Skipped 1 previous similar message Sep 4 10:43:34 cs04r-sc-serv-06 kernel: LustreError: 875:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:43:34 cs04r-sc-serv-06 kernel: LustreError: 875:0:(dir.c:398:ll_get_dir_page()) Skipped 2 previous similar messages Sep 4 10:43:34 cs04r-sc-serv-06 kernel: LustreError: 875:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:43:34 cs04r-sc-serv-06 kernel: LustreError: 875:0:(dir.c:584:ll_dir_read()) Skipped 1 previous similar message Sep 4 10:47:19 cs04r-sc-serv-06 kernel: LustreError: 1122:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0xe900001:0x3b1189d1:0x0] at 0: rc -5 Sep 4 10:47:19 cs04r-sc-serv-06 kernel: LustreError: 1122:0:(dir.c:584:ll_dir_read()) error reading dir [0xe900001:0x3b1189d1:0x0] at 0: rc -5 </pre> </div></div> <p>I'll attach the full MDT syslog file starting with the mount until we unmounted again to fail back to the previous MDT as a file.</p> <p>Note that IB and lnet over IB has been added to this file system recently, following the instructions in the manual on changing server NIDs, including unmounting everything, unloading lustre modules on the servers completely, tunefs.lustre --writeconf --erase-param with the new NIDs etc, mounting MGS, MDT, OSTs, in this order. (Some ethernet only clients might have been still up during this, but the client I used to test this while it wasn't working certainly had been unmounted then and rebooted a few times after).</p> <p>We are currently concerned that this will happen again if we have to do another fail over on the MDT, so want to solve this. Let us know what other information we should provide.</p> RHEL6 server, RHEL6 clients, servers connected to IB and ethernet, clients can be either connected to IB and ethernet or just ethernet LU-5583

clients receive IO error after MDT failover

Bug Major Open Unresolved Zhenyu Xu Frederik Ferner Thu, 4 Sep 2014 12:55:17 +0000 Tue, 7 Jun 2016 15:38:26 +0000 Lustre 2.5.2 0 3 <p>Bobijam</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>from the mds log, it shows that for some unknown reason, the MGS does not work correctly, </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 4 09:55:21 cs04r-sc-mds03-02 kernel: Lustre: MGS: non-config logname received: params Sep 4 09:55:22 cs04r-sc-mds03-02 kernel: Lustre: MGS: non-config logname received: params ... Sep 4 09:56:35 cs04r-sc-mds03-02 kernel: LustreError: 43873:0:(obd_mount_server.c:1136:server_register_target()) lustre03-MDT0000: error registering with the MGS: rc = -5 (not fatal) ... Sep 4 09:57:11 cs04r-sc-mds03-02 kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy. </pre> </div></div> <p>Is MGT is a separate device and is it mounted elsewhere while this MDS node is trying to mount it?</p> <p>Yes, the MGS is a separate partition, it would also be affected by any MDT fail over and our scripts will mount both of them on the same server. Though now that I think about it, I'm not convinced any order is enforced in those scripts. It has always worked so far...</p> <p>so I've just tried again, with the same result at least as far as the logs and the MDT are concerned, this time ignoring any scripts and doing all steps manually after a fresh reboot of the failover MDS. </p> <p>The following steps seem to reproduce it in this setup every time:</p> <ul> <li>umount MDT and MGS on the current MDS</li> <li>mount MGS on the failover MDS, wait a few seconds (tried up to about 1 minute)</li> <li>mount MDT on the same failover MDS, the appears to work.</li> </ul> <p>The same errors about as above are in the logs, though this time all clients that I've tried seem to work and new clients can mount the file system, so the MGS appears to work at least for them.</p> <p>I've then (on the same failover MDS) attempted to unmount/mount the MDT, this fails with the following log messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 9 23:46:26 cs04r-sc-mds03-02 kernel: Lustre: server umount lustre03-MDT0000 complete Sep 9 23:46:37 cs04r-sc-mds03-02 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=off. Opts: Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(genops.c:320:class_newdev()) Device MGC10.144.144.1@o2ib already exists at 4, won't add Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_config.c:374:class_attach()) Cannot create device MGC10.144.144.1@o2ib of type mgc : -17 Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_mount.c:195:lustre_start_simple()) MGC10.144.144.1@o2ib attach error -17 Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_mount_server.c:861:lustre_disconnect_lwp()) lustre03-MDT0000-lwp-MDT0000: Can't end config log lustre03-client. Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_mount_server.c:1436:server_put_super()) lustre03-MDT0000: failed to disconnect lwp. (rc=-2) Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_mount_server.c:1466:server_put_super()) no obd lustre03-MDT0000 Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: LustreError: 40807:0:(obd_mount_server.c:135:server_deregister_mount()) lustre03-MDT0000 not registered Sep 9 23:46:38 cs04r-sc-mds03-02 kernel: Lustre: server umount lustre03-MDT0000 complete </pre> </div></div> <p>Repeating the steps above on the initially active MDS again does not generate the last two log entries you highlighted (only the first two) and cycling through umount/mount for just the MDT works as expected and succeeds in mounting the MDT every time.</p> <p>Looking at tunefs.lustre output (below), I don't see any typo in the IP addresses for the mgs node, but maybe there's another problem, so I'll put them here in case it's relevant and/or helps</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@cs04r-sc-mds03-01 ~]$ sudo tunefs.lustre --print /dev/vg_lustre03/mdt checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: lustre03-MDT0000 Index: 0 Lustre FS: lustre03 Mount type: ldiskfs Flags: 0x1401 (MDT no_primnode ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro,acl Parameters: mgsnode=10.144.144.1@o2ib,172.23.144.1@tcp mgsnode=10.144.144.2@o2ib,172.23.144.2@tcp failover.node=10.144.144.1@o2ib,172.23.144.1@tcp failover.node=10.144.144.2@o2ib,172.23.144.2@tcp mdt.quota_type=ug mdt.group_upcall=/usr/sbin/l_getgroups Permanent disk data: Target: lustre03-MDT0000 Index: 0 Lustre FS: lustre03 Mount type: ldiskfs Flags: 0x1401 (MDT no_primnode ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro,acl Parameters: mgsnode=10.144.144.1@o2ib,172.23.144.1@tcp mgsnode=10.144.144.2@o2ib,172.23.144.2@tcp failover.node=10.144.144.1@o2ib,172.23.144.1@tcp failover.node=10.144.144.2@o2ib,172.23.144.2@tcp mdt.quota_type=ug mdt.group_upcall=/usr/sbin/l_getgroups exiting before disk write. [bnh65367@cs04r-sc-mds03-01 ~]$ sudo tunefs.lustre --print /dev/vg_lustre03/mgs checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: MGS Index: unassigned Lustre FS: lustre Mount type: ldiskfs Flags: 0x4 (MGS ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro Parameters: Permanent disk data: Target: MGS Index: unassigned Lustre FS: lustre Mount type: ldiskfs Flags: 0x4 (MGS ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro Parameters: exiting before disk write. [bnh65367@cs04r-sc-mds03-01 ~]$ </pre> </div></div> <p>Additional help in debugging this would be appreciated.</p> <p>this looks like <a href="https://jira.whamcloud.com/browse/LU-4943" title="Client Failes to mount filesystem" class="issue-link" data-issue-key="LU-4943"><del>LU-4943</del></a> issue (MGC device does not clean up before another mount), would you mind trying patch <a href="http://review.whamcloud.com/#/c/11765/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11765/</a> on the MDS?</p> <p>This patch has fixed the umount/remount issue on the failover MDS. </p> <p>It didn't fix the issue with these entries below, but I don't think you expect this to be fixed by the patch, just stating for clarity</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel: LustreError: 13a-8: Failed to get MGS log params and no local copy. </pre> </div></div> <p>On the other hand, I'm not completely able to reproduce the initial issue anymore, it looks like all clients can talk to the MDT and <tt>lctl get_param lod.lustre03-MDT0000-mdtlov.target_obd</tt> returns all OSTs as active.</p> <p>yes, you are right, #11765 is just for umount/remount issue. The failure to get MGS log I think could be related to the multiple mount of MGT device at the same time.</p> <p>This is how we had been running before the upgrade without any problem and as far as I can see it's only been an issue after the upgrade. Is the recommendation these days to have a separate MGS on a separate machine?</p> <p>Anyway, is the failure to get the MGS log a problem we need to worry about or is it mainly cosmetic?</p> <p>you can use one MGS node for different filesystem, but need separate MGT device for different filesystem. </p> <p>I've done a few more tests on this file system while I can (planned maintenance, nearly over now).</p> <p>I'll try to summarise the results here and hopefully they'll be useful for something (at least they'll be for us to remember what has been tested if we get back to this later..)</p> <p>In this file system, we have one MGT and one MDT, both share the same disk backend and are on the same LVM VG, separate LVs. We have two MDS servers able to access this storage (cs04r-sc-mds03-01 and cs04r-sc-mds03-02), both have lnet configured to use TCP and o2ib. The MDT has is configured to access the MGS on either of the servers, via two mgsnode parameters, both listing o2ib and tcp IP addresses.</p> <p>When the MGT and MDT are mounted in this order on cs04r-sc-mds03-01 all seems to be well, no messages in syslog about failure to get MGS log params or anything.<br/> When the MGT and then the MDT are mounted in this order on cs04r-sc-mds03-02, we get these messages about the failure to get MGS log params but other than the first time, the MDT appears to be working fine.<br/> Mounting the MGT on cs04r-sc-mds03-01 and later mounting the MDT on cs04r-sc-mds03-02 also works fine, no errors in syslog.<br/> Mounting the MGT on cs04r-sc-mds03-02 and later mounting the MDT on cs04r-sc-mds03-01 generates the messages about failure to get MGS log params on cs04r-sc-mds03-01.</p> <p>So, it seems the MGT works on cs04r-sc-mds03-01 but not on cs04r-sc-mds03-02.</p> Related LU-5585 Development End date Wed, 10 Sep 2014 12:55:17 +0000 Rank 1|hzwvbb: Rank (Obsolete) 15574 Severity Start date Thu, 4 Sep 2014 12:55:17 +0000