https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4282 Lustre <p>As indicated in <a href="https://jira.whamcloud.com/browse/LU-4242" title="mdt_open.c:1685:mdt_reint_open()) LBUG" class="issue-link" data-issue-key="LU-4242"><del>LU-4242</del></a>, I now have a problem on our preproduction file system that stops users from accessing the data, servers from cleanly rebooting etc, stopping any further testing.</p> <p>After upgrading the servers from 2.3 to 2.4.1 (MDT build #51 of b2_4 from jenkins) our clients can no longer fully access this file system. The clients can mount the file system and can access one OST on each of the two OSSes, but the other OSSes are not accessible and are shown as inactive in lfs df output and /proc/fs/lustre/lov/*/target_obd, but are shown as UP in lctl dl.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@cs04r-sc-serv-07 ~]$ lctl dl |grep play01 91 UP lov play01-clilov-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 4 92 UP mdc play01-MDT0000-mdc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 93 UP osc play01-OST0000-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 94 UP osc play01-OST0001-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 95 UP osc play01-OST0002-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 96 UP osc play01-OST0003-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 97 UP osc play01-OST0004-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 98 UP osc play01-OST0005-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 [bnh65367@cs04r-sc-serv-07 ~]$ lfs df /mnt/play01 UUID 1K-blocks Used Available Use% Mounted on play01-MDT0000_UUID 78636320 3502948 75133372 4% /mnt/play01[MDT:0] play01-OST0000_UUID 7691221300 4506865920 3184355380 59% /mnt/play01[OST:0] play01-OST0001_UUID 7691221300 3765688064 3925533236 49% /mnt/play01[OST:1] play01-OST0002_UUID : inactive device play01-OST0003_UUID : inactive device play01-OST0004_UUID : inactive device play01-OST0005_UUID : inactive device filesystem summary: 15382442600 8272553984 7109888616 54% /mnt/play01 [bnh65367@cs04r-sc-serv-07 ~]$ cat /proc/fs/lustre/lov/play01-clilov-ffff810076ae2000/target_obd 0: play01-OST0000_UUID ACTIVE 1: play01-OST0001_UUID ACTIVE 2: play01-OST0002_UUID INACTIVE 3: play01-OST0003_UUID INACTIVE 4: play01-OST0004_UUID INACTIVE 5: play01-OST0005_UUID INACTIVE </pre> </div></div> <p>As expected the fail-over OSS for each OST does see connection attempts and reports (correctly) that that OST is not available on this OSS.</p> <p>I have confirmed that the OSTs are mounted on the OSSes correctly.</p> <p>For the other client that I have tried to bring back the situation is similar but the OSTs that are inactive are slightly different:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@cs04r-sc-serv-06 ~]$ lfs df /mnt/play01 UUID 1K-blocks Used Available Use% Mounted on play01-MDT0000_UUID 78636320 3502948 75133372 4% /mnt/play01[MDT:0] play01-OST0000_UUID : inactive device play01-OST0001_UUID 7691221300 3765688064 3925533236 49% /mnt/play01[OST:1] play01-OST0002_UUID 7691221300 1763305508 5927915792 23% /mnt/play01[OST:2] play01-OST0003_UUID : inactive device play01-OST0004_UUID : inactive device play01-OST0005_UUID : inactive device filesystem summary: 15382442600 5528993572 9853449028 36% /mnt/play01 [bnh65367@cs04r-sc-serv-06 ~]$ </pre> </div></div> <p>play01-OST0000, play01-OST0002, play01-OST0004 are on one OSS<br/> play01-OST0001, play01-OST0003, play01-OST0005 are on a different OSS (but all on the same).</p> <p>I have tested the network, don't see any errors, lnet_selftest between the clients and the OSSes works at line rate at least for the first client (1GigE client...), nothing obvious on the second client either.</p> <p>For completeness I should probably mention that all the servers (MDS and OSSes) have changed IP addresses at the same time as the upgrade, I have verified the information is correctly changed on the targets, both clients have been rebooted multiple times since the IP address change, without any changes.</p> MDS and OSS on Lustre 2.4.1, clients lustre 1.8.9, all Red Hat Enterprise Linux. LU-4282

some OSTs reported as inactive in lfs df, UP with lctl dl, data not accessible

Bug Blocker Resolved Duplicate Minh Diep Frederik Ferner Wed, 20 Nov 2013 20:04:33 +0000 Thu, 20 Feb 2014 21:16:25 +0000 Fri, 24 Jan 2014 16:20:48 +0000 Lustre 2.4.1 0 6 <p>Hi Frederik,</p> <p>Could you show us the command you used to change IP address on the servers?</p> <p>I unmounted the MDT, MGT, all OSTs, and the two clients I'm currently trying to use, other clients were left up to be rebooted later, changed the options etc in /etc/modprobe.d/lustre.conf to bring up the correct NIDs on the servers, confirmed with lctl list_nids that the correct IPs were identified, then ran this on the mds/mgs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tunefs.lustre --erase-params --writeconf /dev/vg_play01/mgs tunefs.lustre --erase-params --writeconf --mgsnode=172.23.144.5@tcp0 --mgsnode=172.23.144.6@tcp0 --servicenode=172.23.144.5@tcp0 --servicenode=172.23.144.6@tcp0 --param mdt.quota_type=ug --param mdt.group_upcall=/usr/sbin/l_getgroups --mountfsoptions=iopen_nopriv,user_xattr,errors=remount-ro,acl /dev/vg_play01/mdt </pre> </div></div> <p>On the OSSes I ran the following for each OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tunefs.lustre --erase-params --writeconf --mgsnode=172.23.144.5@tcp0 --mgsnode=172.23.144.6@tcp0 --servicenode=172.23.144.14@tcp0 --servicenode=172.23.144.18@tcp0 --param ost.quota_type=ug /dev/mapper/ost_play01_0 </pre> </div></div> <p>The mounted first MGS, then MGT, then all OSTs, then tried to bring those two clients back... I've actually ran those commands a few more times now, partly because the LBUG on the MDT seemed to confuse things...</p> <p>Why did you include two mgsnode?</p> <p>could you show tunefs.lustre --dryrun &lt;mdt&gt;, and tunefs.lustre --dryrun &lt;ost&gt; ?</p> <p>to mgsnode because the MGS will fail over between the same two machines as the MDT even though it is on a separate partition.</p> <p>tunefs.lustre --dryrun for mdt and ost:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@cs04r-sc-mds02-03 ~]$ sudo tunefs.lustre --dryrun /dev/mapper/vg_play01-mdt checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-MDT0000 Index: 0 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1001 (MDT no_primnode ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro,acl Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.5@tcp failover.node=172.23.144.6@tcp mdt.quota_type=ug mdt.group_upcall=/usr/sbin/l_getgroups Permanent disk data: Target: play01-MDT0000 Index: 0 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1001 (MDT no_primnode ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro,acl Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.5@tcp failover.node=172.23.144.6@tcp mdt.quota_type=ug mdt.group_upcall=/usr/sbin/l_getgroups exiting before disk write. [bnh65367@cs04r-sc-mds02-03 ~]$ </pre> </div></div> <p>OSTs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@cs04r-sc-oss01-04 ~]$ for i in /dev/mapper/ost_play01_* ; do sudo tunefs.lustre --dryrun $i ; done checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0000 Index: 0 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1402 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0000 Index: 0 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1402 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug exiting before disk write. checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0001 Index: 1 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0001 Index: 1 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug exiting before disk write. checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0002 Index: 2 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0002 Index: 2 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug exiting before disk write. checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0003 Index: 3 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0003 Index: 3 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug exiting before disk write. checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0004 Index: 4 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0004 Index: 4 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.14@tcp failover.node=172.23.144.18@tcp ost.quota_type=ug exiting before disk write. checking for existing Lustre data: found Reading CONFIGS/mountdata Read previous values: Target: play01-OST0005 Index: 5 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug Permanent disk data: Target: play01-OST0005 Index: 5 Lustre FS: play01 Mount type: ldiskfs Flags: 0x1002 (OST no_primnode ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=172.23.144.5@tcp mgsnode=172.23.144.6@tcp failover.node=172.23.144.18@tcp failover.node=172.23.144.14@tcp ost.quota_type=ug exiting before disk write. [bnh65367@cs04r-sc-oss01-04 ~]$ </pre> </div></div> <p>I don't understand "MGS will fail over between the same two machines as the MDT even though it is on a separate partition."</p> <p>I assume that cs04r-sc-mds02-03 = 72.23.144.5@tcp</p> <p>and another-mss-host = 172.23.144.6@tcp</p> <p>are these two sharing the same storage/device?</p> <p>Sorry, should have provided a bit more background...</p> <p>The file system has two OSS in active-active configuration, with the new IPs 172.23.144.14 and 172.23.144.18 sharing a storage array. For the MDS we also have two servers sharing a storage array, the new IPs for those are indeed 172.23.144.5 (cs04r-sc-mds02-03) and 172.23.144.6 (cs04r-sc-mds02-04).</p> <p>Cheers,<br/> Frederik</p> <p>ok thanks. Ah, I also see you share mgs? or it's a typo?</p> <p>tunefs.lustre --erase-params --writeconf /dev/vg_play01/mgs &lt;&lt;&lt;&lt;&lt;<br/> tunefs.lustre --erase-params --writeconf --mgsnode=172.23.144.5@tcp0 --mgsnode=172.23.144.6@tcp0 --servicenode=172.23.144.5@tcp0 --servicenode=172.23.144.6@tcp0 --param mdt.quota_type=ug --param mdt.group_upcall=/usr/sbin/l_getgroups --mountfsoptions=iopen_nopriv,user_xattr,errors=remount-ro,acl /dev/vg_play01/mdt</p> <p>If you have combined mgs/mdt, then you should only have 1 mgsnode<br/> tunefs.lustre --erase-params --writeconf /dev/vg_play01/mdt<br/> tunefs.lustre --writeconf --mgsnode=172.23.144.6@tcp0 --servicenode=172.23.144.5@tcp0 --servicenode=172.23.144.6@tcp0 --param mdt.quota_type=ug --param mdt.group_upcall=/usr/sbin/l_getgroups --mountfsoptions=iopen_nopriv,user_xattr,errors=remount-ro,acl --mgs --mdt /dev/vg_play01/mdt</p> <p>Note: no --erase-params on second tunefs.lustre cmd</p> <p>don't forget the unmount all clients and OST while you --writeconf the MDS,</p> <p>Then run<br/> tunefs.lustre --writeconf --ost /dev/mapper/ost_play01_0</p> <p>The MGS is on the same shared storage as the MDT, same LVM volume group but a separate logical volume.</p> <p>So I think I need your first set of commands. though I don't see how they are much different from mine. In any case, I've run them again after umounting everything, brought everything back up, no change.</p> <p>This time I noticed the following -16 errors in the logs, I assume they are because the OSTs are still in recovery but thought I'd mention them. Also there is this initially error about communication with 0@lo that I don't recall seeing before.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: Lustre: MGS: Logs for fs play01 were removed by user request. All servers must be restarted in order to regenerate the logs. Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: Lustre: Setting parameter play01-MDT0000.mdt.quota_type in log play01-MDT0000 Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: Lustre: Skipped 1 previous similar message Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: Lustre: play01-MDT0000: used disk, loading Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: Lustre: 4012:0:(mdt_handler.c:4948:mdt_process_config()) For interoperability, skip this mdt.quota_type. It is obsolete. Nov 21 17:30:06 cs04r-sc-mds02-03 kernel: LustreError: 11-0: play01-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. Nov 21 17:30:32 cs04r-sc-mds02-03 kernel: Lustre: MGS: Regenerating play01-OST0000 log by user request. Nov 21 17:30:32 cs04r-sc-mds02-03 kernel: Lustre: Setting parameter play01-OST0000.ost.quota_type in log play01-OST0000 Nov 21 17:30:32 cs04r-sc-mds02-03 kernel: Lustre: Skipped 1 previous similar message Nov 21 17:30:38 cs04r-sc-mds02-03 kernel: Lustre: MGS: Regenerating play01-OST0001 log by user request. Nov 21 17:30:38 cs04r-sc-mds02-03 kernel: Lustre: Setting parameter play01-OST0001.ost.quota_type in log play01-OST0001 Nov 21 17:30:39 cs04r-sc-mds02-03 kernel: LustreError: 11-0: play01-OST0000-osc-MDT0000: Communicating with 172.23.144.14@tcp, operation ost_connect failed with -16. Nov 21 17:30:54 cs04r-sc-mds02-03 kernel: Lustre: MGS: Regenerating play01-OST0002 log by user request. Nov 21 17:30:54 cs04r-sc-mds02-03 kernel: Lustre: Setting parameter play01-OST0002.ost.quota_type in log play01-OST0002 Nov 21 17:31:02 cs04r-sc-mds02-03 kernel: LustreError: 11-0: play01-OST0003-osc-MDT0000: Communicating with 172.23.144.18@tcp, operation ost_connect failed with -16. Nov 21 17:31:02 cs04r-sc-mds02-03 kernel: LustreError: Skipped 1 previous similar message Nov 21 17:31:07 cs04r-sc-mds02-03 kernel: Lustre: MGS: Regenerating play01-OST0004 log by user request. Nov 21 17:31:07 cs04r-sc-mds02-03 kernel: Lustre: Skipped 1 previous similar message Nov 21 17:31:10 cs04r-sc-mds02-03 kernel: LustreError: 11-0: play01-OST0001-osc-MDT0000: Communicating with 172.23.144.18@tcp, operation ost_connect failed with -16. Nov 21 17:31:11 cs04r-sc-mds02-03 kernel: Lustre: Setting parameter play01-OST0005.ost.quota_type in log play01-OST0005 Nov 21 17:31:11 cs04r-sc-mds02-03 kernel: Lustre: Skipped 2 previous similar messages Nov 21 17:31:44 cs04r-sc-mds02-03 kernel: LustreError: 11-0: play01-OST0000-osc-MDT0000: Communicating with 172.23.144.14@tcp, operation ost_connect failed with -16. Nov 21 17:31:44 cs04r-sc-mds02-03 kernel: LustreError: Skipped 2 previous similar messages Nov 21 17:32:34 cs04r-sc-mds02-03 kernel: Lustre: play01-MDT0000: Will be in recovery for at least 5:00, or until 1 client reconnects Nov 21 17:32:34 cs04r-sc-mds02-03 kernel: Lustre: play01-MDT0000: Denying connection for new client 45cd72fa-56c3-f257-0ed7-154d629ee603 (at 172.23.136.7@tcp), waiting for all 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 4:59 </pre> </div></div> <p>Ok, more testing with this showed that for all OSTs that were shown as inactive on the clients, the fail over OSS was seeing connection attempts. So failing over the OSTs to the other node makes them available on the clients, however failing over OSTs that were active makes them unavailable on the clients.</p> <p>So to make this clearer:</p> <p>Initially the OSTs were distributed like this:</p> <p>OSTs mounted on 172.23.144.14: play01-OST0000, play01-OST0002, play01-OST0004<br/> OSTs mounted on 172.23.144.18: play01-OST0001, play01-OST0003, play01-OST0005</p> <p>In this configuration the clients were only able to access play01-OST0000 and play01-OST0001.</p> <p>The following distribution of OSTs makes them available on both clients I tested today:</p> <p>OSTs mounted on 172.23.144.14: play01-OST0000, play01-OST0003, play01-OST0005<br/> OSTs mounted on 172.23.144.18: play01-OST0001, play01-OST0002, play01-OST0004</p> <p>As soon as any of the OSTs is mounted on the other OSS, none of the clients will connect to it, it appears (with a possible exception of clients that have not been rebooted recently, unloading lustre modules and starting again on the client seems to bring them into the first category.</p> <p>The same parameters/failnode setup works without problems so far on our other file systems were all servers are still running 1.8.</p> <p>Hi Frederik,</p> <p>Is it working now? I believe there might be a small step that we missed somewhere during the process. Please let me know if everything is working. Thanks</p> <p>Minh,</p> <p>it is sort of working. I have one configuration/setup where all OSTs can be accessed by all clients I've tried to bring up. However if I try to bring any of the OSTs up on a different OSS then they are on now, none of my clients even tries to contact this OSS. Recovery doesn't even start...</p> <p>So I'd not say everything is working, but the urgency is lower as we have a work around. (which is valid until one of the servers fails...)</p> <p>I would appreciate help in fully resolving this. Let me know if there are any diagnostics that I should provide...</p> <p>Kind regards,<br/> Frederik</p> <p>This seems to relate to <a href="https://jira.whamcloud.com/browse/LU-4243" title="multiple servicenodes or failnids: wrong client llog registration" class="issue-link" data-issue-key="LU-4243"><del>LU-4243</del></a>. could you remount the mdt with ldiskfs and dump the config log?</p> <p>CONFIGS directories for MDT and MGS, including llog_reader output</p> <p>Minh,</p> <p>I wasn't quite sure which logs you want, so I remounted both mdt and mgs with ldiskfs, copied all files in the CONFIGS directories to a different location and ran llog_reader over them. The result is in the attached file.</p> <p>Thanks,<br/> Frederik</p> <p>Hongchao,</p> <p>Could you check if this is a dump of <a href="https://jira.whamcloud.com/browse/LU-4243" title="multiple servicenodes or failnids: wrong client llog registration" class="issue-link" data-issue-key="LU-4243"><del>LU-4243</del></a>? Thanks</p> <p>I looked at the client log</p> <p>Header size : 8192<br/> Time : Thu Nov 21 20:51:25 2013<br/> Number of records: 21<br/> Target uuid : play01-client <br/> -----------------------<br/> #01 (224)marker 4 (flags=0x01, v2.4.1.0) play01-clilov 'lov setup' Thu Nov 21 20:51:25 2013-<br/> #02 (120)attach 0:play01-clilov 1:lov 2:play01-clilov_UUID <br/> #03 (168)lov_setup 0:play01-clilov 1:(struct lov_desc)<br/> uuid=play01-clilov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1<br/> #04 (224)marker 4 (flags=0x02, v2.4.1.0) play01-clilov 'lov setup' Thu Nov 21 20:51:25 2013-<br/> #05 (224)marker 5 (flags=0x01, v2.4.1.0) play01-clilmv 'lmv setup' Thu Nov 21 20:51:25 2013-<br/> #06 (120)attach 0:play01-clilmv 1:lmv 2:play01-clilmv_UUID <br/> #07 (168)lov_setup 0:play01-clilmv 1:(struct lov_desc)<br/> uuid=play01-clilmv_UUID stripe:cnt=0 size=0 offset=0 pattern=0<br/> #08 (224)marker 5 (flags=0x02, v2.4.1.0) play01-clilmv 'lmv setup' Thu Nov 21 20:51:25 2013-<br/> #09 (224)marker 6 (flags=0x01, v2.4.1.0) play01-MDT0000 'add mdc' Thu Nov 21 20:51:25 2013-<br/> #10 (088)add_uuid nid=172.23.144.5@tcp(0x20000ac179005) 0: 1:172.23.144.5@tcp <br/> #11 (128)attach 0:play01-MDT0000-mdc 1:mdc 2:play01-clilmv_UUID <br/> #12 (144)setup 0:play01-MDT0000-mdc 1:play01-MDT0000_UUID 2:172.23.144.5@tcp <br/> #13 (088)add_uuid nid=172.23.144.5@tcp(0x20000ac179005) 0: 1:172.23.144.5@tcp <br/> #14 (112)add_conn 0:play01-MDT0000-mdc 1:172.23.144.5@tcp <br/> #15 (088)add_uuid nid=172.23.144.6@tcp(0x20000ac179006) 0: 1:172.23.144.5@tcp <br/> #16 (112)add_conn 0:play01-MDT0000-mdc 1:172.23.144.5@tcp <br/> #17 (160)modify_mdc_tgts add 0:play01-clilmv 1:play01-MDT0000_UUID 2:0 3:1 4:play01-MDT0000-mdc_UUID <br/> #18 (224)marker 6 (flags=0x02, v2.4.1.0) play01-MDT0000 'add mdc' Thu Nov 21 20:51:25 2013-<br/> #19 (224)marker 7 (flags=0x01, v2.4.1.0) play01-client 'mount opts' Thu Nov 21 20:51:25 2013-<br/> #20 (120)mount_option 0: 1:play01-client 2:play01-clilov 3:play01-clilmv <br/> #21 (224)marker 7 (flags=0x02, v2.4.1.0) play01-client 'mount opts' Thu Nov 21 20:51:25 2013-</p> <p>line #15 add_uuid should have 172.23.144.6 on the second nid instead of *5.</p> <p>This is a dup of <a href="https://jira.whamcloud.com/browse/LU-4243" title="multiple servicenodes or failnids: wrong client llog registration" class="issue-link" data-issue-key="LU-4243"><del>LU-4243</del></a> IMHO</p> <p>yes, It should be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4243" title="multiple servicenodes or failnids: wrong client llog registration" class="issue-link" data-issue-key="LU-4243"><del>LU-4243</del></a>.</p> <p>Hi Frederik, could you please try with the patch <a href="http://review.whamcloud.com/#/c/8372/?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8372/?</a> thanks</p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-4243" title="multiple servicenodes or failnids: wrong client llog registration" class="issue-link" data-issue-key="LU-4243"><del>LU-4243</del></a></p> <p>Frederick &#8211; can I check if this is now resolved? If so, I will mark it as such. Thanks ~ jfc.</p> <p>Frederick &#8211; my error. I see this is already resolved, so no action required. ~ jfc.</p> Related LU-4242 Development Rank 1|hzw9tj: Rank (Obsolete) 11756 Severity