Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1287] MDS refuses client connections https://jira.whamcloud.com/browse/LU-1287 Lustre <p>When configuring a lustre filesystem with the following setup, the MDS infinitely refuses client connections:</p> <ul class="alternate" type="square"> <li>one MDT alone on a node</li> <li>one or more OSTs on other nodes</li> <li>separate MGS</li> <li>filesystem started in the following order (OSTs, MDT)</li> </ul> <p>Once the filesystem is started, the 'mount' command on the client stays blocked forever.</p> <p>Please find attached a log from the MDS node.</p> LU-1287

MDS refuses client connections

Bug Major Closed Fixed Lai Siyao Florent Thery Thu, 5 Apr 2012 11:27:12 +0000 Wed, 27 Feb 2013 00:32:42 +0000 Wed, 27 Feb 2013 00:31:14 +0000 Lustre 2.4.0 0 4 <p>There does not appear to be a log attached, please provide more details. </p> <p>The Lustre kernel log (lctl dk) is not especially helpful in these situations. Please examine your system logs, (typically /var/log/messages) on all the servers and the impacted clients, there should be some more helpful messages there. Are you certain the servers are all up cleanly? The log mentions a writeconf - did you restart all servers? </p> <p>Actually, the issue arises even when not using the writeconf option.<br/> The issue has been reproduced from scratch, e.g.:</p> <ul class="alternate" type="square"> <li>mkfs the MGS</li> <li>mkfs the filesystem</li> <li>mount the MGS</li> <li>mount the OSTs</li> <li>mount the MDT</li> <li>try to mount the client</li> </ul> <p>Attached is a tarball containing the dmesg output on each node.<br/> Thanks.</p> <p>We've been investigating this issue today to figure out exactly in which case it arises.<br/> We've found that bugzilla #24050 relates to the same issue.<br/> So we've experienced a problem of start order between the MDT and OSTs.<br/> Starting the MDT before the OSTs when it is the first time fixes the issue.</p> <p>We have a number of questions though:</p> <ul class="alternate" type="square"> <li>will this issue be fixed on the lustre code level ?</li> <li>is there a section addressing this situation in the lustre manual ?<br/> (Section 13.2 speaks about start orders but does not seem to address the situation)</li> <li>on the user level, is there an easy way to troubleshoot this situation (e.g. a log line that helps to identify, ...) ?</li> </ul> <p>Thanks<br/> Florent.</p> <p>I am sorry, but your logs are extremely short. You say you have re-created the filesystem, I see no logs showing this.<br/> I do see the MDS is refusing the connection: Lustre: b10-MDT0000: temporarily refusing client connection from 60.64.2.24@o2ib<br/> but threre really should be more detail there. Please attach the output of tunefs.lustre --print <device> for the MGT, MDT and all OSTs. I would suggest a full cold start of the systems, verifying system health at each step.</p> <p>What is your state? Do you have more information on this issue? </p> <p>Hi,</p> <p>From a full cold start system running Lustre 2.1, I reran the test by doing sequentially:</p> <p>1. format MGS</p> <p>At this point, 'tunefs.lustre --print' on the MGT gives:</p> <p><span class="error">[root@perou2 ~]</span># tunefs.lustre --print /dev/disk/by-id/scsi-2003013841aac0025<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: mgs<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: failover.node=10.3.1.2@o2ib</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: mgs<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: failover.node=10.3.1.2@o2ib</p> <p>exiting before disk write.</p> <p>2. start MGS</p> <p>3. format MDT</p> <p>At this point, 'tunefs.lustre --print' on the MDT gives:</p> <p><span class="error">[root@perou3 ~]</span># tunefs.lustre --print /dev/disk/by-id/scsi-2003013841aac002d<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: fs_mdt-MDT0000<br/> Index: 0<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x61<br/> (MDT first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: mgsnode=10.5.1.3@o2ib lov.stripesize=1048576 failover.node=10.5.1.3@o2ib network=o2ib0</p> <p> Permanent disk data:<br/> Target: fs_mdt-MDT0000<br/> Index: 0<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x61<br/> (MDT first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: mgsnode=10.5.1.3@o2ib lov.stripesize=1048576 failover.node=10.5.1.3@o2ib network=o2ib0</p> <p>exiting before disk write.</p> <p>4. format OSTs</p> <p>At this point, 'tunefs.lustre --print' on the OSTs gives:</p> <p><span class="error">[root@perou6 ~]</span># tunefs.lustre --print /dev/disk/by-id/scsi-2003013841aac0037<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: fs_mdt-OST0000<br/> Index: 0<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x62<br/> (OST first_time update )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=10.5.1.3@o2ib failover.node=10.5.1.6@o2ib network=o2ib0</p> <p> Permanent disk data:<br/> Target: fs_mdt-OST0000<br/> Index: 0<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x62<br/> (OST first_time update )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=10.5.1.3@o2ib failover.node=10.5.1.6@o2ib network=o2ib0</p> <p>exiting before disk write.</p> <p><span class="error">[root@perou6 ~]</span># tunefs.lustre --print /dev/disk/by-id/scsi-2003013841aac0035<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: fs_mdt-OST0001<br/> Index: 1<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x62<br/> (OST first_time update )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=10.5.1.3@o2ib failover.node=10.5.1.6@o2ib network=o2ib0</p> <p> Permanent disk data:<br/> Target: fs_mdt-OST0001<br/> Index: 1<br/> Lustre FS: fs_mdt<br/> Mount type: ldiskfs<br/> Flags: 0x62<br/> (OST first_time update )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=10.5.1.3@o2ib failover.node=10.5.1.6@o2ib network=o2ib0</p> <p>exiting before disk write.</p> <p>5. start OSTs</p> <p>At that point, here is the content of the MGS:</p> <p><span class="error">[root@perou2 ~]</span># ls toto/CONFIGS/<br/> fs_mdt-client fs_mdt-OST0000 fs_mdt-OST0001 fs_mdt-sptlrpc _mgs-sptlrpc mountdata<br/> <span class="error">[root@perou2 ~]</span># llog_reader toto/CONFIGS/fs_mdt-OST0000<br/> Header size : 8192<br/> Time : Fri Apr 13 15:07:03 2012<br/> Number of records: 4<br/> Target uuid : config_uuid<br/> -----------------------<br/> #01 (224)marker 5 (flags=0x01, v2.1.0.0) fs_mdt-OST0000 'add ost' Fri Apr 13 15:07:03 2012-<br/> #02 (128)attach 0:fs_mdt-OST0000 1:obdfilter 2:fs_mdt-OST0000_UUID<br/> #03 (112)setup 0:fs_mdt-OST0000 1:dev 2:type 3:f<br/> #04 (224)marker 5 (flags=0x02, v2.1.0.0) fs_mdt-OST0000 'add ost' Fri Apr 13 15:07:03 2012-<br/> <span class="error">[root@perou2 ~]</span>#<br/> <span class="error">[root@perou2 ~]</span># llog_reader toto/CONFIGS/fs_mdt-OST0001<br/> Header size : 8192<br/> Time : Fri Apr 13 15:07:03 2012<br/> Number of records: 4<br/> Target uuid : config_uuid<br/> -----------------------<br/> #01 (224)marker 1 (flags=0x01, v2.1.0.0) fs_mdt-OST0001 'add ost' Fri Apr 13 15:07:03 2012-<br/> #02 (128)attach 0:fs_mdt-OST0001 1:obdfilter 2:fs_mdt-OST0001_UUID<br/> #03 (112)setup 0:fs_mdt-OST0001 1:dev 2:type 3:f<br/> #04 (224)marker 1 (flags=0x02, v2.1.0.0) fs_mdt-OST0001 'add ost' Fri Apr 13 15:07:03 2012-<br/> <span class="error">[root@perou2 ~]</span>#</p> <p>6. start MDT</p> <p>At that point, the content of the MGS is:</p> <p><span class="error">[root@perou2 ~]</span># ls toto/CONFIGS/<br/> fs_mdt-client fs_mdt-MDT0000 fs_mdt-OST0000 fs_mdt-OST0001 fs_mdt-sptlrpc _mgs-sptlrpc mountdata<br/> <span class="error">[root@perou2 ~]</span>#<br/> <span class="error">[root@perou2 ~]</span># llog_reader toto/CONFIGS/fs_mdt-OST0000<br/> Header size : 8192<br/> Time : Fri Apr 13 15:07:03 2012<br/> Number of records: 4<br/> Target uuid : config_uuid<br/> -----------------------<br/> #01 (224)marker 5 (flags=0x01, v2.1.0.0) fs_mdt-OST0000 'add ost' Fri Apr 13 15:07:03 2012-<br/> #02 (128)attach 0:fs_mdt-OST0000 1:obdfilter 2:fs_mdt-OST0000_UUID<br/> #03 (112)setup 0:fs_mdt-OST0000 1:dev 2:type 3:f<br/> #04 (224)marker 5 (flags=0x02, v2.1.0.0) fs_mdt-OST0000 'add ost' Fri Apr 13 15:07:03 2012-<br/> <span class="error">[root@perou2 ~]</span>#<br/> <span class="error">[root@perou2 ~]</span># llog_reader toto/CONFIGS/fs_mdt-OST0001<br/> Header size : 8192<br/> Time : Fri Apr 13 15:07:03 2012<br/> Number of records: 4<br/> Target uuid : config_uuid<br/> -----------------------<br/> #01 (224)marker 1 (flags=0x01, v2.1.0.0) fs_mdt-OST0001 'add ost' Fri Apr 13 15:07:03 2012-<br/> #02 (128)attach 0:fs_mdt-OST0001 1:obdfilter 2:fs_mdt-OST0001_UUID<br/> #03 (112)setup 0:fs_mdt-OST0001 1:dev 2:type 3:f<br/> #04 (224)marker 1 (flags=0x02, v2.1.0.0) fs_mdt-OST0001 'add ost' Fri Apr 13 15:07:03 2012-</p> <p>And in the syslog of the MDS node we have:</p> <p>1334322067 2012 Apr 13 15:01:07 perou3 kern warning kernel LDISKFS-fs warning (device sdau): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> 1334322068 2012 Apr 13 15:01:08 perou3 kern info kernel LDISKFS-fs (sdau): barriers disabled<br/> 1334322068 2012 Apr 13 15:01:08 perou3 kern info kernel LDISKFS-fs (sdau): mounted filesystem with ordered data mode<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel LDISKFS-fs warning (device sdau): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel LDISKFS-fs (sdau): barriers disabled<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel LDISKFS-fs (sdau): mounted filesystem with ordered data mode<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel LDISKFS-fs warning (device sdau): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel LDISKFS-fs (sdau): barriers disabled<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel LDISKFS-fs (sdau): mounted filesystem with ordered data mode<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: 7750:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC10.5.1.3@o2ib->MGC10.5.1.3@o2ib_0 netid 50000: select flavor null<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: MGC10.5.1.3@o2ib: Reactivating import<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel Lustre: Enabling ACL<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern info kernel Lustre: Enabling user_xattr<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: fs_mdt-MDT0000: new disk, initializing<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: 7783:0:(mds_lov.c:1004:mds_notify()) MDS mdd_obd-fs_mdt-MDT0000: add target fs_mdt-OST0001_UUID<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: 7783:0:(mds_lov.c:1004:mds_notify()) Skipped 1 previous similar message<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800069 sent from fs_mdt-OST0001-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: [<br/> sent 1334322545] <span class="error">[real_sent 1334322545]</span> <span class="error">[current 1334322545]</span> <span class="error">[deadline 5s]</span> <span class="error">[delay -5s]</span> req@ffff88030a0dc000 x1398844479800069/t0(0) o-1->fs_mdt-OST0001_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322550 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322545 2012 Apr 13 15:09:05 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800070 sent from fs_mdt-OST0000-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: <span class="error">[sent 1334322545]</span> <span class="error">[real_sent 1334322545]</span> <span class="error">[current 1334322545]</span> <span class="error">[deadline 5s]</span> <span class="error">[delay -5s]</span> req@ffff88030a0f0000 x1398844479800070/t0(0) o-1->fs_mdt-OST0000_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322550 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322570 2012 Apr 13 15:09:30 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 5s<br/> 1334322570 2012 Apr 13 15:09:30 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800072 sent from fs_mdt-OST0001-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: <span class="error">[sent 1334322570]</span> <span class="error">[real_sent 1334322570]</span> <span class="error">[current 1334322570]</span> <span class="error">[deadline 10s]</span> <span class="error">[delay -10s]</span> req@ffff8803313d3400 x1398844479800072/t0(0) o-1->fs_mdt-OST0001_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322580 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322577 2012 Apr 13 15:09:37 perou3 kern warning kernel Lustre: 619:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800071 sent from MGC10.5.1.3@o2ib to NID 10.5.1.3@o2ib has timed out for slow reply: <span class="error">[sent 1334322570]</span> <span class="error">[real_sent 1334322570]</span> <span class="error">[current 1334322577]</span> <span class="error">[deadline 7s]</span> <span class="error">[delay 0s]</span> req@ffff88031fdf4000 x1398844479800071/t0(0) o-1->MGS@MGC10.5.1.3@o2ib_0:26/25 lens 192/192 e 0 to 1 dl 1334322577 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322577 2012 Apr 13 15:09:37 perou3 kern warning kernel Lustre: 619:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br/> 1334322577 2012 Apr 13 15:09:37 perou3 kern err kernel LustreError: 166-1: MGC10.5.1.3@o2ib: Connection to service MGS via nid 10.5.1.3@o2ib was lost; in progress operations using this service will fail.<br/> 1334322583 2012 Apr 13 15:09:43 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800074 sent from MGC10.5.1.3@o2ib to NID 10.5.1.3@o2ib has timed out for slow reply: <span class="error">[sent 1334322577]</span> <span class="error">[real_sent 1334322577]</span> <span class="error">[current 1334322583]</span> <span class="error">[deadline 6s]</span> <span class="error">[delay 0s]</span> req@ffff88031fdf4000 x1398844479800074/t0(0) o-1->MGS@MGC10.5.1.3@o2ib_0:26/25 lens 368/512 e 0 to 1 dl 1334322583 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322602 2012 Apr 13 15:10:02 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) MGC10.5.1.3@o2ib: tried all connections, increasing latency to 6s<br/> 1334322602 2012 Apr 13 15:10:02 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 1 previous similar message<br/> 1334322602 2012 Apr 13 15:10:02 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800076 sent from fs_mdt-OST0001-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: <span class="error">[sent 1334322602]</span> <span class="error">[real_sent 1334322602]</span> <span class="error">[current 1334322602]</span> <span class="error">[deadline 15s]</span> <span class="error">[delay -15s]</span> req@ffff88017cafa800 x1398844479800076/t0(0) o-1->fs_mdt-OST0001_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322617 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322613 2012 Apr 13 15:10:13 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800075 sent from MGC10.5.1.3@o2ib to NID 10.5.1.3@o2ib has timed out for slow reply: <span class="error">[sent 1334322602]</span> <span class="error">[real_sent 1334322602]</span> <span class="error">[current 1334322613]</span> <span class="error">[deadline 11s]</span> <span class="error">[delay 0s]</span> req@ffff88030a074800 x1398844479800075/t0(0) o-1->MGS@MGC10.5.1.3@o2ib_0:26/25 lens 368/512 e 0 to 1 dl 1334322613 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322613 2012 Apr 13 15:10:13 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br/> 1334322627 2012 Apr 13 15:10:27 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) MGC10.5.1.3@o2ib: tried all connections, increasing latency to 11s<br/> 1334322627 2012 Apr 13 15:10:27 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 2 previous similar messages<br/> 1334322643 2012 Apr 13 15:10:43 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800078 sent from MGC10.5.1.3@o2ib to NID 10.5.1.3@o2ib has timed out for slow reply: <span class="error">[sent 1334322627]</span> <span class="error">[real_sent 1334322627]</span> <span class="error">[current 1334322643]</span> <span class="error">[deadline 16s]</span> <span class="error">[delay 0s]</span> req@ffff88032908c000 x1398844479800078/t0(0) o-1->MGS@MGC10.5.1.3@o2ib_0:26/25 lens 368/512 e 0 to 1 dl 1334322643 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322643 2012 Apr 13 15:10:43 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br/> 1334322652 2012 Apr 13 15:10:52 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) MGC10.5.1.3@o2ib: tried all connections, increasing latency to 16s 1334322652 2012 Apr 13 15:10:52 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 1 previous similar message<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) MGC10.5.1.3@o2ib: tried all connections, increasing latency to 21s<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 2 previous similar messages<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800084 sent from fs_mdt-OST0001-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: <span class="error">[sent 1334322677]</span> <span class="error">[real_sent 1334322677]</span> <span class="error">[current 1334322677]</span> <span class="error">[deadline 30s]</span> <span class="error">[delay -30s]</span> req@ffff8801c35a6c00 x1398844479800084/t0(0) o-1->fs_mdt-OST0001_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322707 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 3 previous similar messages<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: 620:0:(import.c:852:ptlrpc_connect_interpret()) MGS@MGC10.5.1.3@o2ib_0 changed server handle from 0x555b88f8bfb49318 to 0x555b88f8bfb49373<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern warning kernel Lustre: MGC10.5.1.3@o2ib: Reactivating import<br/> 1334322677 2012 Apr 13 15:11:17 perou3 kern info kernel Lustre: MGC10.5.1.3@o2ib: Connection restored to service MGS using nid 10.5.1.3@o2ib.<br/> 1334322702 2012 Apr 13 15:11:42 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 30s<br/> 1334322702 2012 Apr 13 15:11:42 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 2 previous similar messages<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 35s<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern err kernel LustreError: 11-0: an error occurred while communicating with 10.5.1.3@o2ib. The obd_ping operation failed with -107<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern err kernel LustreError: 166-1: MGC10.5.1.3@o2ib: Connection to service MGS via nid 10.5.1.3@o2ib was lost; in progress operations using this service will fail.<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern warning kernel Lustre: 620:0:(import.c:852:ptlrpc_connect_interpret()) MGS@MGC10.5.1.3@o2ib_0 changed server handle from 0x555b88f8bfb49373 to 0x555b88f8bfb493dc<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern warning kernel Lustre: MGC10.5.1.3@o2ib: Reactivating import<br/> 1334322727 2012 Apr 13 15:12:07 perou3 kern info kernel Lustre: MGC10.5.1.3@o2ib: Connection restored to service MGS using nid 10.5.1.3@o2ib.<br/> 1334322752 2012 Apr 13 15:12:32 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 40s<br/> 1334322752 2012 Apr 13 15:12:32 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 1 previous similar message<br/> 1334322752 2012 Apr 13 15:12:32 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1398844479800103 sent from fs_mdt-OST0001-osc-MDT0000 to NID 10.5.1.6@o2ib has failed due to network error: <span class="error">[sent 1334322752]</span> <span class="error">[real_sent 1334322752]</span> <span class="error">[current 1334322752]</span> <span class="error">[deadline 45s]</span> <span class="error">[delay -45s]</span> req@ffff88030a0dc800 x1398844479800103/t0(0) o-1->fs_mdt-OST0001_UUID@10.5.1.5@o2ib:28/4 lens 368/512 e 0 to 1 dl 1334322797 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1334322752 2012 Apr 13 15:12:32 perou3 kern warning kernel Lustre: 620:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 4 previous similar messages<br/> 1334322802 2012 Apr 13 15:13:22 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 50s<br/> 1334322802 2012 Apr 13 15:13:22 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 2 previous similar messages<br/> 1334322877 2012 Apr 13 15:14:37 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) fs_mdt-OST0001-osc-MDT0000: tried all connections, increasing latency to 50s<br/> 1334322877 2012 Apr 13 15:14:37 perou3 kern warning kernel Lustre: 621:0:(import.c:526:import_select_connection()) Skipped 5 previous similar messages</p> <p>On the MDS, we can see:<br/> <span class="error">[root@perou3 ~]</span># lctl dl<br/> 0 UP mgc MGC10.5.1.3@o2ib f2d4e47f-96b1-e539-f0e1-e125d27e617f 5<br/> 1 UP lov fs_mdt-MDT0000-mdtlov fs_mdt-MDT0000-mdtlov_UUID 4<br/> 2 UP mdt fs_mdt-MDT0000 fs_mdt-MDT0000_UUID 3<br/> 3 UP mds mdd_obd-fs_mdt-MDT0000 mdd_obd_uuid-fs_mdt-MDT0000 3<br/> 4 UP osc fs_mdt-OST0001-osc-MDT0000 fs_mdt-MDT0000-mdtlov_UUID 5<br/> 5 UP osc fs_mdt-OST0000-osc-MDT0000 fs_mdt-MDT0000-mdtlov_UUID 5</p> <p>7. Mount client</p> <p>In the client syslog, we have:</p> <p>1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: OBD class driver, <a href="http://wiki.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/</a><br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Lustre Version: 2.1.0<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Build Version: B-2_1_0_0-lustrebull-20120404161806-CHANGED-2.6.32-71.24.1.bl6.Bull.23.x86_64<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Lustre LU module (ffffffffa053c2c0).<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Added LNI 10.5.1.6@o2ib <span class="error">[8/64/0/180]</span><br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Lustre OSC module (ffffffffa09780c0).<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Lustre LOV module (ffffffffa09e3e40).<br/> 1336046968 2012 May 3 14:09:28 perou7 kern info kernel Lustre: Lustre client module (ffffffffa0d392e0).<br/> 1336046968 2012 May 3 14:09:28 perou7 user info logger lustre-tune: 0 devices have been tuned.<br/> 1336046968 2012 May 3 14:09:28 perou7 kern warning kernel Lustre: 21809:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC10.5.1.3@o2ib->MGC10.5.1.3@o2ib_0 netid 50000: select flavor null<br/> 1336046968 2012 May 3 14:09:28 perou7 kern err kernel LustreError: 152-6: Ignoring deprecated mount option 'acl'.<br/> 1336046968 2012 May 3 14:09:28 perou7 kern warning kernel Lustre: MGC10.5.1.3@o2ib: Reactivating import<br/> 1336046968 2012 May 3 14:09:28 perou7 kern warning kernel Lustre: 21809:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import fs_mdt-MDT0000-mdc-ffff8802d12c9000->10.5.1.4@o2ib netid 50000: select flavor null<br/> 1336046968 2012 May 3 14:09:28 perou7 kern err kernel LustreError: 11-0: an error occurred while communicating with 10.5.1.4@o2ib. The mds_connect operation failed with -11<br/> 1336046993 2012 May 3 14:09:53 perou7 kern err kernel LustreError: 11-0: an error occurred while communicating with 10.5.1.4@o2ib. The mds_connect operation failed with -11<br/> 1336047018 2012 May 3 14:10:18 perou7 kern err kernel LustreError: 11-0: an error occurred while communicating with 10.5.1.4@o2ib. The mds_connect operation failed with -11</p> <p>At the same time, in the MDS log we have:</p> <p>1336046968 2012 May 3 14:09:28 perou3 kern warning kernel Lustre: fs_mdt-MDT0000: temporarily refusing client connection from 10.5.1.6@o2ib<br/> 1336046968 2012 May 3 14:09:28 perou3 kern err kernel LustreError: 26684:0:(ldlm_lib.c:2137:target_send_reply_msg()) @@@ processing error (<del>11) req@ffff88032beafc00 x1400946785517577/t0(0) o-1</del>><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1336047068 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1<br/> 1336046993 2012 May 3 14:09:53 perou3 kern warning kernel Lustre: fs_mdt-MDT0000: temporarily refusing client connection from 10.5.1.6@o2ib<br/> 1336046993 2012 May 3 14:09:53 perou3 kern err kernel LustreError: 26684:0:(ldlm_lib.c:2137:target_send_reply_msg()) @@@ processing error (<del>11) req@ffff88032bee3000 x1400946785517580/t0(0) o-1</del>><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1336047093 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1</p> <p>This issue looks like <a href="https://jira.whamcloud.com/browse/LU-350" title="port bug24050 to master("lustre_start" caused client nodes failed to mount.)" class="issue-link" data-issue-key="LU-350"><del>LU-350</del></a>, but the fact is the fix from this ticket is landed into Lustre 2.1.<br/> The wierd thing here is that when the MDT is started, it tries to reach the failover node of the OSTs (NID 10.5.1.6@o2ib) and apparently not their primary node.<br/> Of course, when starting the MDT before the OSTs, the MDT connects directly to the OSTs with the right NID, ie the primary one.</p> <p>Regards,<br/> Sebastien.</p> <p>Lai</p> <p>Could you please analyze this situation?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Florent, what's the output of `llog_reader toto/CONFIGS/fs_mdt-MDT0000` after you started MDS? The nid of OST (MDS connects to) should be written in this config. Could you print this config for the case of MDS starting first also?</p> <p>Hi,</p> <p>When we start OSTs first and then MDT (case when we hit the bug):</p> <ol> <li>llog_reader toto/CONFIGS/fs_mdt-MDT0000<br/> Header size : 8192<br/> Time : Wed May 9 13:08:57 2012<br/> Number of records: 32<br/> Target uuid : config_uuid <br/> -----------------------<br/> #01 (224)marker 7 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:08:57 2012-<br/> #02 (136)attach 0:fs_mdt-MDT0000-mdtlov 1:lov 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #03 (176)lov_setup 0:fs_mdt-MDT0000-mdtlov 1:(struct lov_desc)<br/> uuid=fs_mdt-MDT0000-mdtlov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1<br/> #04 (224)marker 7 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:08:57 2012-<br/> #05 (224)marker 8 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:08:57 2012-<br/> #06 (120)attach 0:fs_mdt-MDT0000 1:mdt 2:fs_mdt-MDT0000_UUID <br/> #07 (112)mount_option 0: 1:fs_mdt-MDT0000 2:fs_mdt-MDT0000-mdtlov <br/> #08 (160)setup 0:fs_mdt-MDT0000 1:fs_mdt-MDT0000_UUID 2:0 3:fs_mdt-MDT0000-mdtlov 4:f <br/> #09 (224)marker 8 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:08:57 2012-<br/> #10 (224)marker 9 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000 'add osc(copied)' Wed May 9 13:08:57 2012-<br/> #11 (224)marker 10 (flags=0x01, v2.1.0.0) fs_mdt-OST0000 'add osc' Wed May 9 13:08:57 2012-<br/> #12 (080)add_uuid nid=10.5.1.5@o2ib(0x500000a050105) 0: 1:10.5.1.5@o2ib <br/> #13 (080)add_uuid nid=10.5.1.6@o2ib(0x500000a050106) 0: 1:10.5.1.5@o2ib <br/> #14 (144)attach 0:fs_mdt-OST0000-osc-MDT0000 1:osc 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #15 (144)setup 0:fs_mdt-OST0000-osc-MDT0000 1:fs_mdt-OST0000_UUID 2:10.5.1.5@o2ib <br/> #16 (136)lov_modify_tgts add 0:fs_mdt-MDT0000-mdtlov 1:fs_mdt-OST0000_UUID 2:0 3:1 <br/> #17 (224)marker 10 (flags=0x02, v2.1.0.0) fs_mdt-OST0000 'add osc' Wed May 9 13:08:57 2012-<br/> #18 (224)marker 10 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000 'add osc(copied)' Wed May 9 13:08:57 2012-<br/> #19 (224)marker 11 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000 'add osc(copied)' Wed May 9 13:08:57 2012-<br/> #20 (224)marker 12 (flags=0x01, v2.1.0.0) fs_mdt-OST0001 'add osc' Wed May 9 13:08:57 2012-<br/> #21 (080)add_uuid nid=10.5.1.5@o2ib(0x500000a050105) 0: 1:10.5.1.5@o2ib <br/> #22 (080)add_uuid nid=10.5.1.6@o2ib(0x500000a050106) 0: 1:10.5.1.5@o2ib <br/> #23 (080)add_uuid nid=10.5.1.5@o2ib(0x500000a050105) 0: 1:10.5.1.5@o2ib <br/> #24 (080)add_uuid nid=10.5.1.6@o2ib(0x500000a050106) 0: 1:10.5.1.5@o2ib <br/> #25 (144)attach 0:fs_mdt-OST0001-osc-MDT0000 1:osc 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #26 (144)setup 0:fs_mdt-OST0001-osc-MDT0000 1:fs_mdt-OST0001_UUID 2:10.5.1.5@o2ib <br/> #27 (136)lov_modify_tgts add 0:fs_mdt-MDT0000-mdtlov 1:fs_mdt-OST0001_UUID 2:1 3:1 <br/> #28 (224)marker 12 (flags=0x02, v2.1.0.0) fs_mdt-OST0001 'add osc' Wed May 9 13:08:57 2012-<br/> #29 (224)marker 12 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000 'add osc(copied)' Wed May 9 13:08:57 2012-<br/> #30 (224)marker 15 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:08:57 2012-<br/> #31 (112)param 0:fs_mdt-MDT0000-mdtlov 1:lov.stripesize=1048576 <br/> #32 (224)marker 15 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:08:57 2012-</li> </ol> <p>When we start MDT first and then the OSTs (no bug):</p> <ul class="alternate" type="square"> <li>right after MDT start:</li> </ul> <ol> <li>llog_reader toto/CONFIGS/fs_mdt-MDT0000<br/> Header size : 8192<br/> Time : Wed May 9 13:21:35 2012<br/> Number of records: 12<br/> Target uuid : config_uuid <br/> -----------------------<br/> #01 (224)marker 1 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:21:35 2012-<br/> #02 (136)attach 0:fs_mdt-MDT0000-mdtlov 1:lov 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #03 (176)lov_setup 0:fs_mdt-MDT0000-mdtlov 1:(struct lov_desc)<br/> uuid=fs_mdt-MDT0000-mdtlov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1<br/> #04 (224)marker 1 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:21:35 2012-<br/> #05 (224)marker 2 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:21:35 2012-<br/> #06 (120)attach 0:fs_mdt-MDT0000 1:mdt 2:fs_mdt-MDT0000_UUID <br/> #07 (112)mount_option 0: 1:fs_mdt-MDT0000 2:fs_mdt-MDT0000-mdtlov <br/> #08 (160)setup 0:fs_mdt-MDT0000 1:fs_mdt-MDT0000_UUID 2:0 3:fs_mdt-MDT0000-mdtlov 4:f <br/> #09 (224)marker 2 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:21:35 2012-<br/> #10 (224)marker 7 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:21:35 2012-<br/> #11 (112)param 0:fs_mdt-MDT0000-mdtlov 1:lov.stripesize=1048576 <br/> #12 (224)marker 7 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:21:35 2012-</li> </ol> <ul class="alternate" type="square"> <li>then after OSTs start:</li> </ul> <ol> <li>llog_reader toto/CONFIGS/fs_mdt-MDT0000<br/> Header size : 8192<br/> Time : Wed May 9 13:21:35 2012<br/> Number of records: 28<br/> Target uuid : config_uuid <br/> -----------------------<br/> #01 (224)marker 1 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:21:35 2012-<br/> #02 (136)attach 0:fs_mdt-MDT0000-mdtlov 1:lov 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #03 (176)lov_setup 0:fs_mdt-MDT0000-mdtlov 1:(struct lov_desc)<br/> uuid=fs_mdt-MDT0000-mdtlov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1<br/> #04 (224)marker 1 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov setup' Wed May 9 13:21:35 2012-<br/> #05 (224)marker 2 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:21:35 2012-<br/> #06 (120)attach 0:fs_mdt-MDT0000 1:mdt 2:fs_mdt-MDT0000_UUID <br/> #07 (112)mount_option 0: 1:fs_mdt-MDT0000 2:fs_mdt-MDT0000-mdtlov <br/> #08 (160)setup 0:fs_mdt-MDT0000 1:fs_mdt-MDT0000_UUID 2:0 3:fs_mdt-MDT0000-mdtlov 4:f <br/> #09 (224)marker 2 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000 'add mdt' Wed May 9 13:21:35 2012-<br/> #10 (224)marker 7 (flags=0x01, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:21:35 2012-<br/> #11 (112)param 0:fs_mdt-MDT0000-mdtlov 1:lov.stripesize=1048576 <br/> #12 (224)marker 7 (flags=0x02, v2.1.0.0) fs_mdt-MDT0000-mdtlov 'lov.stripesize' Wed May 9 13:21:35 2012-<br/> #13 (224)marker 10 (flags=0x01, v2.1.0.0) fs_mdt-OST0001 'add osc' Wed May 9 13:22:26 2012-<br/> #14 (080)add_uuid nid=10.5.1.5@o2ib(0x500000a050105) 0: 1:10.5.1.5@o2ib <br/> #15 (144)attach 0:fs_mdt-OST0001-osc-MDT0000 1:osc 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #16 (144)setup 0:fs_mdt-OST0001-osc-MDT0000 1:fs_mdt-OST0001_UUID 2:10.5.1.5@o2ib <br/> #17 (080)add_uuid nid=10.5.1.6@o2ib(0x500000a050106) 0: 1:10.5.1.6@o2ib <br/> #18 (112)add_conn 0:fs_mdt-OST0001-osc-MDT0000 1:10.5.1.6@o2ib <br/> #19 (136)lov_modify_tgts add 0:fs_mdt-MDT0000-mdtlov 1:fs_mdt-OST0001_UUID 2:1 3:1 <br/> #20 (224)marker 10 (flags=0x02, v2.1.0.0) fs_mdt-OST0001 'add osc' Wed May 9 13:22:26 2012-<br/> #21 (224)marker 13 (flags=0x01, v2.1.0.0) fs_mdt-OST0000 'add osc' Wed May 9 13:22:27 2012-<br/> #22 (080)add_uuid nid=10.5.1.5@o2ib(0x500000a050105) 0: 1:10.5.1.5@o2ib <br/> #23 (144)attach 0:fs_mdt-OST0000-osc-MDT0000 1:osc 2:fs_mdt-MDT0000-mdtlov_UUID <br/> #24 (144)setup 0:fs_mdt-OST0000-osc-MDT0000 1:fs_mdt-OST0000_UUID 2:10.5.1.5@o2ib <br/> #25 (080)add_uuid nid=10.5.1.6@o2ib(0x500000a050106) 0: 1:10.5.1.6@o2ib <br/> #26 (112)add_conn 0:fs_mdt-OST0000-osc-MDT0000 1:10.5.1.6@o2ib <br/> #27 (136)lov_modify_tgts add 0:fs_mdt-MDT0000-mdtlov 1:fs_mdt-OST0000_UUID 2:0 3:1 <br/> #28 (224)marker 13 (flags=0x02, v2.1.0.0) fs_mdt-OST0000 'add osc' Wed May 9 13:22:27 2012-</li> </ol> <p>For the record, the OSS hosting the 2 OSTs has NID 10.5.1.5@o2ib.</p> <p>Cheers,<br/> Sebastien.</p> <p>The llog config looks fine. And I did the same test, but can't reproduce with master code. Could you `lctl set_param debug=-1` on MDS node, and dump debug log after MDS mounted. I'll test 2.1 later.</p> <p>2.1 test can pass in my test env too.</p> <p>Hi,</p> <p>Here is the requested debug information:</p> <ul class="alternate" type="square"> <li>lustre_mds_ko.gz is the debug log when the OSTs are mounted first, which means we git the error;</li> <li>lustre_mds_ok.gz is the debug log when the MDT is mounted first, so that we have no error.</li> </ul> <p>HTH<br/> Sebastien.</p> <p>It looks like the config generated for MDS(for the failure case) is wrong: both nid 10.5.1.5@o2ib and 10.5.1.6@o2ib are mapped to uuid 10.5.1.5@o2ib, and during MDS osc connect, it tends to use the last nid, which is 10.5.1.6@o2ib, thus connection always fail. I'll dig into config code to see why this happens.</p> <p>Once OSTs start before MDT, MDT needs steal OSC config from client config, however both target nids and failover nids are written in config as the same type: LCFG_ADD_UUID. Previously, all these nids are treated as target nids, so wrong config file is generated for MDT. And ptlrpc tends to use the last nid of target (if all nids in the same subnet) to connect to target, so the error happens.</p> <p>The patch for master is at: <a href="http://review.whamcloud.com/#change,3107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3107</a>.</p> <p>It should be able to be patched to 2.1 branch. Sebastien, could you help verify that it works for you?</p> <p>Hi,</p> <p>I gave a try to the patch.<br/> The good news is it fixes the issue when we mount OSTs before the MDT, having the 'first_time' ldd flag set (meaning it is the first time this Lustre file system is mounted).<br/> But the bad news is it does not fix the issue when we explicitly set the 'writeconf' flag (and same starting order: OSTs then MDT).</p> <p>On the MGS, the logs are:</p> <ul class="alternate" type="square"> <li>when we start OSTs:</li> </ul> <p>1339769487 2012 Jun 15 16:11:27 perou2 kern warning kernel Lustre: 956:0:(ldlm_lib.c:876:target_handle_connect()) MGS: connection from 527e8357-0c75-491a-738c-18026d6ec94c@10.5.1.5@o2ib t0 exp (null) cur 1339769487 last 0<br/> 1339769487 2012 Jun 15 16:11:27 perou2 kern warning kernel Lustre: 956:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x500000a050105_UUID netid 50000: select flavor null<br/> 1339769487 2012 Jun 15 16:11:27 perou2 kern warning kernel Lustre: 955:0:(ldlm_lib.c:791:target_handle_connect()) MGS: exp ffff8802eb2ad800 already connecting<br/> 1339769487 2012 Jun 15 16:11:27 perou2 kern err kernel LustreError: 955:0:(mgs_handler.c:783:mgs_handle()) MGS handle cmd=250 rc=-114<br/> 1339769487 2012 Jun 15 16:11:27 perou2 kern err kernel LustreError: 955:0:(ldlm_lib.c:2137:target_send_reply_msg()) @@@ processing error (<del>114) req@ffff8803179b0850 x1404849096753257/t0(0) o-1</del>><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1339769587 ref 1 fl Interpret:/ffffffff/ffffffff rc -114/-1<br/> 1339769512 2012 Jun 15 16:11:52 perou2 kern warning kernel Lustre: MGS: Regenerating fs_mdt-OST0001 log by user request.</p> <ul class="alternate" type="square"> <li>when we then start the MDT:</li> </ul> <p>1339769542 2012 Jun 15 16:12:22 perou2 kern warning kernel Lustre: 956:0:(ldlm_lib.c:876:target_handle_connect()) MGS: connection from 35c02d1f-5920-b87f-bc64-358fbcf00f05@10.5.1.4@o2ib t0 exp (null) cur 1339769542 last 0<br/> 1339769542 2012 Jun 15 16:12:22 perou2 kern warning kernel Lustre: 956:0:(ldlm_lib.c:876:target_handle_connect()) Skipped 1 previous similar message<br/> 1339769542 2012 Jun 15 16:12:22 perou2 kern warning kernel Lustre: MGS: Logs for fs fs_mdt were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> 1339769542 2012 Jun 15 16:12:22 perou2 kern info kernel Lustre: Setting parameter fs_mdt-MDT0000-mdtlov.lov.stripesize in log fs_mdt-MDT0000<br/> 1339769542 2012 Jun 15 16:12:22 perou2 kern info kernel Lustre: Skipped 1 previous similar message</p> <p>So half of the problem is solved ;-(</p> <p>HTH,<br/> Sebastien.</p> <p>Sebastien, could you tell me how did you set 'writeconf'? In my understanding 'writeconf' implies removing all existed config for the specified fs, so in your test, when MDT started, it removes all configs for 'fs_mdt' (including OSTs). Could you explain why you set 'writeconf', and normally how you use it?</p> <p>Hi,</p> <p>I use 'writeconf' when I need to reformat a file system.</p> <p>For instance, consider I have a 'fs1' file system formated and running, talking to my MGS. Now I stop this 'fs1' file system, and build a new file system (different OSTs, and/or different MDT) while keeping the same 'fs1' name. Then, if I format this new 'fs1' file system, there will be a problem on the MGS, because the configuration information for 'fs1' will be there already. This is why I need to use the 'writeconf' parameter.</p> <p>I always set the 'writeconf' parameter on the OSTs as well as the MDT:</p> <ol> <li>mkfs.lustre --reformat --fsname=fs_mdt --mdt --index=0 --mgsnode=perou2-ib0@o2ib0 --param=lov.stripesize=1048576 --network=o2ib0 --writeconf --mkfsoptions="-j -J device=/dev/disk/by-id/scsi-2003013841aac000d -m 0" /dev/disk/by-id/scsi-2003013841aac002d</li> <li>mkfs.lustre --reformat --fsname=fs_mdt --ost --index=0 --mgsnode=perou2-ib0@o2ib0 --failnode=perou7-ib0@o2ib0 --network=o2ib0 --writeconf --mkfsoptions="-j -J device=/dev/disk/by-id/scsi-2003013841aac0017 -m 0" /dev/disk/by-id/scsi-2003013841aac0037</li> <li>mkfs.lustre --reformat --fsname=fs_mdt --ost --index=1 --mgsnode=perou2-ib0@o2ib0 --failnode=perou7-ib0@o2ib0 --network=o2ib0 --writeconf --mkfsoptions="-j -J device=/dev/disk/by-id/scsi-2003013841aac0015 -m 0" /dev/disk/by-id/scsi-2003013841aac0035</li> </ol> <p>If I set 'writeconf' on all targets, I do not understand why the configuration should not be regenerated on the MGS, even if I start the OSTs first.</p> <p>Cheers,<br/> Sebastien.</p> <p>Lustre Manual 14.4 Regenerating Lustre Configuration Logs:<br/> Run the writeconf command on all servers (MDT first, then OSTs)<br/> Start the file system in this order:<br/> MGS (or the combined MGS/MDT)<br/> MDT<br/> OSTs<br/> Lustre clients</p> <p>And the code also shows that writeconf MDT will erase all fs log (MDT, OSTs and clients), so there is message like this in your log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1339769542 2012 Jun 15 16:12:22 perou2 kern warning kernel Lustre: MGS: Logs for fs fs_mdt were removed by user request. All servers must be restarted in order to regenerate the logs. </pre> </div></div> <p>In my understanding this is a special case, and I'm not clear how 'writeconf' is originally designed, but this doesn't look to be an issue.</p> <p>Hum, sorry. You are absolutely right, one is supposed to invert target start order when 'writeconf' flag is set.</p> <p>So, in the end, I think I can say your patch is working great!<br/> I have changed my review to '+1' in <a href="http://review.whamcloud.com/3107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3107</a> <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Cheers,<br/> Sebastien.</p> <p>Patch landed.</p> Development Rank 1|hzvqa7: Rank (Obsolete) 8134 Severity