Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-4120] Not a bug file system unavailable. https://jira.whamcloud.com/browse/LU-4120 Lustre <p>We have had a file system down to be upgraded since Monday.</p> <p>We can not bring the filesystem in to operation.</p> <p>We have been talking with DDN support since Tuesday.</p> ubuntu 12.04 LTS LU-4120

Not a bug file system unavailable.

Bug Major Resolved Not a Bug Bob Glossman James Beal Fri, 18 Oct 2013 13:52:00 +0000 Tue, 28 Oct 2014 20:46:23 +0000 Tue, 28 Oct 2014 20:46:23 +0000 Lustre 1.8.9 Lustre 2.4.1 0 8 <p>The system was using the default e2fsprogs until very recently. We have now installed 1.42.7.wc1 with the patch to get the quota working.</p> <p>The system originally had both of the servers it could be on listed as failnode rather than just the other node in the pair.</p> <p>At one point we had 26 out of 30 OSS's available but the last 4 would not come on line ( this was after only those 4 were working ).</p> <p>We have replaced the kernel modules and user space tools after <a href="https://jira.whamcloud.com/browse/LU-4111" title="client kernel panic" class="issue-link" data-issue-key="LU-4111"><del>LU-4111</del></a>, however this configuration has worked previously on three other filesystems.</p> <p>At present when we mount an OST it does not become up in the MDT's lctl device list but stays in AT eg.</p> <p>root@lus04-mds1:~# lctl dl<br/> 0 UP mgs MGS MGS 7<br/> 1 UP mgc MGC172.17.148.4@tcp 3f2155e6-d9ad-156f-d431-406faeb78ef5 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 AT osc lus04-OST0000-osc lus04-mdtlov_UUID 1</p> <p>We have downgraded the system to 1.8.9wc1 and the system performs in the same way.</p> <p>MGS/MDT mount logs<br/> Oct 18 14:29:06 lus04-mds1 kernel: [ 7549.035620] Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> Oct 18 14:29:06 lus04-mds1 kernel: [ 7549.081427] Lustre: Added LNI 172.17.148.4@tcp <span class="error">[8/256/0/180]</span><br/> Oct 18 14:29:06 lus04-mds1 kernel: [ 7549.081520] Lustre: Accept secure, port 988<br/> Oct 18 14:29:06 lus04-mds1 kernel: [ 7549.135974] Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.145508] LDISKFS-fs (dm-1): barriers disabled<br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.147409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.261222] LDISKFS-fs (dm-1): barriers disabled<br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.262804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.289975] Lustre: MGS MGS started<br/> Oct 18 14:29:07 lus04-mds1 kernel: [ 7549.290696] Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.133538] LDISKFS-fs (dm-0): barriers disabled<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.136647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.265245] LDISKFS-fs (dm-0): barriers disabled<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.267566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.298400] LustreError: 13c-e: Client log lus04-client has disappeared! Regenerating all logs.<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.298589] Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.313147] Lustre: Enabling user_xattr<br/> Oct 18 14:29:16 lus04-mds1 kernel: [ 7558.343685] Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/mapper/lus04--mdt0-lus04 with recovery enabled<br/> Oct 18 14:29:47 lus04-mds1 kernel: [ 7590.037063] Lustre: MGS: Regenerating lus04-OST0000 log by user request.<br/> Oct 18 14:29:51 lus04-mds1 kernel: [ 7593.422391] LustreError: 21629:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> Oct 18 14:29:51 lus04-mds1 kernel: [ 7593.422561] LustreError: 21629:0:(obd_config.c:372:class_setup()) setup lus04-OST0000-osc failed (-2)<br/> Oct 18 14:29:51 lus04-mds1 kernel: [ 7593.422695] LustreError: 21629:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> Oct 18 14:29:51 lus04-mds1 kernel: [ 7593.422832] Lustre: cmd=cf003 0:lus04-OST0000-osc 1:lus04-OST0000_UUID 2:0@<0:0> </p> <p>OSS mount logs<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.183166]</span> Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.229796]</span> Lustre: Added LNI 172.17.148.6@tcp <span class="error">[8/256/0/180]</span><br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.229842]</span> Lustre: Accept secure, port 988<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.285325]</span> Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.370273]</span> LDISKFS-fs (dm-14): barriers disabled<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.392095]</span> LDISKFS-fs (dm-14): mounted filesystem with ordered data mode<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.578277]</span> LDISKFS-fs (dm-14): barriers disabled<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.587805]</span> LDISKFS-fs (dm-14): mounted filesystem with ordered data mode<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.601175]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 18 14:29:47 lus04-oss1 kernel: <span class="error">[11025.644539]</span> Lustre: Filtering OBD driver; <a href="http://wiki.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/</a><br/> Oct 18 14:29:48 lus04-oss1 kernel: <span class="error">[11026.007326]</span> Lustre: lus04-OST0000: Now serving lus04-OST0000 on /dev/mapper/vd00 with recovery enabled<br/> root@lus04-oss1:/# cat /proc/fs/lustre/obdfilter/lus04-OST0000/recovery_status <br/> status: INACTIVE</p> <p>Bob,<br/> Could you please have a look at this one?<br/> Thank you!</p> <p>Looking into possible causes. I suspect it may be a side effect of upgrading/downgrading between 1.8.9 and 2.4.1. Hope to have more and better information to give you soon.</p> <p>Bob we had the same symptoms when the system was running 2.4.1</p> <p>I can try and generate a log of the discussion with DDN and email or work out how to attach it as a private file.</p> <p>what state is your fs in now? all umounted? all mounted but some OSTs inactive? just MDT mounted?</p> <p>We have the MGS and MDT mounted and a single OSS.</p> <p>We have stopped as lctl dl shows on the MGS/MDT.</p> <p> 0 UP mgs MGS MGS 7<br/> 1 UP mgc MGC172.17.148.4@tcp 3f2155e6-d9ad-156f-d431-406faeb78ef5 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 AT osc lus04-OST0000-osc lus04-mdtlov_UUID 1</p> <p>Note the OSS is stuck in AT not up.</p> <p>Additionally when we try and mount the file system from am client</p> <p>root@isg-disc-mon-05:~# dmesg<br/> <span class="error">[127708.147231]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> <span class="error">[127708.149699]</span> LustreError: 2312:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[127708.149912]</span> LustreError: 2312:0:(obd_config.c:372:class_setup()) setup lus04-MDT0000-mdc-ffff8806181bcc00 failed (-2)<br/> <span class="error">[127708.150204]</span> LustreError: 2312:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> <span class="error">[127708.151628]</span> Lustre: cmd=cf003 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> <span class="error">[127708.151957]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> <span class="error">[127708.152374]</span> LustreError: 2302:0:(llite_lib.c:1099:ll_fill_super()) Unable to process log: -2<br/> <span class="error">[127708.152695]</span> LustreError: 2302:0:(obd_config.c:443:class_cleanup()) Device 2 not setup<br/> <span class="error">[127708.152975]</span> LustreError: 2302:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> <span class="error">[127708.153170]</span> LustreError: 2302:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> <span class="error">[127708.155666]</span> Lustre: client lus04-client(ffff8806181bcc00) umount complete<br/> <span class="error">[127708.155791]</span> LustreError: 2302:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-2)</p> <p>A couple of things to try and verify. Some of these may be repeats of things you've already done.</p> <p>Confirm you have the same version of e2fsprogs on all your servers.</p> <p>Unmount everything.</p> <p>Bring up at least lnet everywhere with 'modprobe lnet', should already be loaded on nodes where you have attempted mounts. get the list of nids on every server with 'lctl list_nids'. see if all the nids shown look real and sensible. The error msg Lustre: cmd=cf003 0:lus04-OST0000-osc 1:lus04-OST0000_UUID 2:0@<0:0> in particular suggests something wrong with nids. 2:0 doesn't look like a sensible value.</p> <p>Examine the config log with llog_reader.<br/> Mount the MGS with -t ldiskfs.<br/> do 'llog_reader <mountpoint>/CONFIGS/$FSNAME-client' where $FSNAME in your fs name (lus04?).<br/> ummount it again.</p> <p>Do a fresh round of tunefs.lustre --writeconf on all your MDTs and OSTs.</p> <p>Try mounting again, MGS/MDT first, then OSTs in index order.</p> <p>Worst case if we can't repair this fs would it be possible to reformat? Seems like the most drastic solution, but most likely to work.</p> <p>"A couple of things to try and verify. Some of these may be repeats of things you've already done.<br/> Confirm you have the same version of e2fsprogs on all your servers."</p> <p>lus04-mds1: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-mds1: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013<br/> lus04-mds2: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-mds2: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013<br/> lus04-oss1: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-oss1: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013<br/> lus04-oss2: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-oss2: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013<br/> lus04-oss3: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-oss3: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013<br/> lus04-oss4: e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lus04-oss4: Using EXT2FS Library version 1.42.7.wc1, 12-Apr-2013</p> <p>One thing I have noticed is that I can't list the nids if I only modprobe lnet rather than lustre.</p> <p>oot@it-admin:~# dsh -M -g lus04 "modprobe lnet"<br/> root@it-admin:~# dsh -M -g lus04 "lctl list_nids"<br/> lus04-mds1: 172.17.148.4@tcp<br/> lus04-mds2: IOC_LIBCFS_GET_NI error 100: Network is down<br/> lus04-oss1: 172.17.148.6@tcp<br/> lus04-oss2: IOC_LIBCFS_GET_NI error 100: Network is down<br/> lus04-oss3: IOC_LIBCFS_GET_NI error 100: Network is down<br/> lus04-oss4: IOC_LIBCFS_GET_NI error 100: Network is down<br/> root@it-admin:~# dsh -M -g lus04 "modprobe lustre"<br/> root@it-admin:~# dsh -M -g lus04 "lctl list_nids"<br/> lus04-mds1: 172.17.148.4@tcp<br/> lus04-mds2: 172.17.148.5@tcp<br/> lus04-oss1: 172.17.148.6@tcp<br/> lus04-oss2: 172.17.148.7@tcp<br/> lus04-oss3: 172.17.148.8@tcp<br/> lus04-oss4: 172.17.148.9@tcp</p> <p>llog_reader CONFIGS/lus04-client <br/> Header size : 8192<br/> Time : Fri Oct 18 14:29:16 2013<br/> Number of records: 18<br/> Target uuid : config_uuid <br/> -----------------------<br/> #01 (224)marker 3 (flags=0x01, v1.8.9.0) lus04-clilov 'lov setup' Fri Oct 18 14:29:16 2013-<br/> #02 (120)attach 0:lus04-clilov 1:lov 2:lus04-clilov_UUID <br/> #03 (168)lov_setup 0:lus04-clilov 1:(struct lov_desc)<br/> uuid=lus04-clilov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1<br/> #04 (224)marker 3 (flags=0x02, v1.8.9.0) lus04-clilov 'lov setup' Fri Oct 18 14:29:16 2013-<br/> #05 (224)marker 4 (flags=0x01, v1.8.9.0) lus04-MDT0000 'add mdc' Fri Oct 18 14:29:16 2013-<br/> #06 (128)attach 0:lus04-MDT0000-mdc 1:mdc 2:lus04-MDT0000-mdc_UUID <br/> #07 (128)setup 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> #08 (088)add_uuid nid=172.17.148.5@tcp(0x20000ac119405) 0: 1:172.17.148.5@tcp <br/> #09 (112)add_conn 0:lus04-MDT0000-mdc 1:172.17.148.5@tcp <br/> #10 (128)mount_option 0: 1:lus04-client 2:lus04-clilov 3:lus04-MDT0000-mdc <br/> #11 (224)marker 4 (flags=0x02, v1.8.9.0) lus04-MDT0000 'add mdc' Fri Oct 18 14:29:16 2013-<br/> #12 (224)marker 7 (flags=0x01, v1.8.9.0) lus04-OST0000 'add osc' Fri Oct 18 14:29:47 2013-<br/> #13 (128)attach 0:lus04-OST0000-osc 1:osc 2:lus04-clilov_UUID <br/> #14 (128)setup 0:lus04-OST0000-osc 1:lus04-OST0000_UUID 2:0@<0:0> <br/> #15 (088)add_uuid nid=172.17.148.7@tcp(0x20000ac119407) 0: 1:172.17.148.7@tcp <br/> #16 (112)add_conn 0:lus04-OST0000-osc 1:172.17.148.7@tcp <br/> #17 (128)lov_modify_tgts add 0:lus04-clilov 1:lus04-OST0000_UUID 2:0 3:1 <br/> #18 (224)marker 7 (flags=0x02, v1.8.9.0) lus04-OST0000 'add osc' Fri Oct 18 14:29:47 2013-</p> <p>I will do the "tunefs.lustre --writeconf" and report back later.</p> <p>While the filesystem has not got irreplaceable information and could be regenerated, the file system was in production. And we would prefer that we could copy the data off to another system before reformatting it.</p> <p>Oh, sorry. I left out a step. After doing 'modprobe lnet' do 'lctl net up'. That should enable the following lctl cmd to work.</p> <p>"Oh, sorry. I left out a step. After doing 'modprobe lnet' do 'lctl net up'. That should enable the following lctl cmd to work."</p> <p>root@it-admin:~# dsh -M -g lus04 "lustre_rmmod"<br/> root@it-admin:~# dsh -M -g lus04 "modprobe lnet"<br/> root@it-admin:~# dsh -M -g lus04 "lctl net up"<br/> lus04-mds1: LNET configured<br/> lus04-mds2: LNET configured<br/> lus04-oss1: LNET configured<br/> lus04-oss2: LNET configured<br/> lus04-oss3: LNET configured<br/> lus04-oss4: LNET configured<br/> root@it-admin:~# dsh -M -g lus04 "lctl list_nids"<br/> lus04-mds1: 172.17.148.4@tcp<br/> lus04-mds2: 172.17.148.5@tcp<br/> lus04-oss1: 172.17.148.6@tcp<br/> lus04-oss2: 172.17.148.7@tcp<br/> lus04-oss3: 172.17.148.8@tcp<br/> lus04-oss4: 172.17.148.9@tcp</p> <p>I will start off with the "tunefs.lustre --writeconf " You say MDTs and OSS's do you want the MGS done as well ?</p> <p>yes, please, all devices. MGS as well. You have a separate MGS and MDT? I was just assuming the MGS/MDT was a combo device.</p> <p>Commands run and sample output for MGS/MDT/OSS writeconf</p> <p>tunefs.lustre --writeconf --mgs --failnode 172.17.148.5 /dev/lus04-mgs0/lus04<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: failover.node=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: failover.node=172.17.148.5@tcp failover.node=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --failnode=172.17.148.5@tcp /dev/lus04-mdt0/lus04</p> <p>Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1001<br/> (MDT no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1141<br/> (MDT update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd15<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd17<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd19<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd21<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd23<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd25<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd27<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.8 --ost /dev/mapper/vd29</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd16<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd18<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd20<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd22<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd24<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd26<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.9 --ost /dev/mapper/vd28</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd01<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd03<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd05<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd07<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd09<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd11<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6 --ost /dev/mapper/vd13</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd00<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd02<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd04<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd06<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd08<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd10<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd12<br/> tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.7 --ost /dev/mapper/vd14</p> <p>Sample output</p> <p>checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04=OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1102<br/> (OST writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.7@tcp</p> <p> Permanent disk data:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.8@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>Possibly need a second opinion, but those params don't look at all right to me. I see 2 --mgsnode options + a --failnode option. Think there should be only 1 --mdsnode + ! --failnode.</p> <p>Just for simplification purposes could you try just setting 1 --mgsnode and no --failnode? I understand this probably isn't your desired target config, but in the interest of getting the fs back up to possibly copy files off simpler is better. If we can get that to work we can work on turning failover back on again later.</p> <p>oot@lus04-mds1:~# dmesg -c<br/> root@lus04-mds1:~# mount -t lustre /dev/lus04-mgs0/lus04 /export/MGS<br/> root@lus04-mds1:~# mount -t lustre /dev/lus04-mdt0/lus04 /export/MDT0<br/> root@lus04-mds1:~# dmesg<br/> <span class="error">[38471.319550]</span> Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> <span class="error">[38471.415542]</span> Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> <span class="error">[38471.424833]</span> LDISKFS-fs (dm-1): barriers disabled<br/> <span class="error">[38471.426341]</span> LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> <span class="error">[38471.524900]</span> LDISKFS-fs (dm-1): barriers disabled<br/> <span class="error">[38471.526415]</span> LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> <span class="error">[38471.553022]</span> Lustre: MGS MGS started<br/> <span class="error">[38471.553767]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> <span class="error">[38484.993123]</span> LDISKFS-fs (dm-0): barriers disabled<br/> <span class="error">[38484.996208]</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> <span class="error">[38485.108837]</span> LDISKFS-fs (dm-0): barriers disabled<br/> <span class="error">[38485.111140]</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> <span class="error">[38485.142781]</span> Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> <span class="error">[38485.156338]</span> Lustre: Enabling user_xattr<br/> <span class="error">[38485.170944]</span> Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/mapper/lus04--mdt0-lus04 with recovery enabled<br/> root@lus04-mds1:~# </p> <p>And the kernel logfile of the server with the MGS and MDT</p> <p><span class="error">[38634.935325]</span> Lustre: MGS: Regenerating lus04-OST0000 log by user request.<br/> <span class="error">[38639.574055]</span> LustreError: 12608:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38639.574234]</span> LustreError: 12608:0:(obd_config.c:372:class_setup()) setup lus04-OST0000-osc failed (-2)<br/> <span class="error">[38639.574391]</span> LustreError: 12608:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> <span class="error">[38639.574563]</span> Lustre: cmd=cf003 0:lus04-OST0000-osc 1:lus04-OST0000_UUID 2:0@<0:0> <br/> <span class="error">[38644.876011]</span> Lustre: MGS: Regenerating lus04-OST0001 log by user request.<br/> <span class="error">[38648.094895]</span> Lustre: MGS: Regenerating lus04-OST0002 log by user request.<br/> <span class="error">[38649.236649]</span> LustreError: 12609:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38649.236820]</span> Lustre: 12609:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0000_UUID<br/> <span class="error">[38649.246930]</span> LustreError: 12609:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0000_UUID not set up<br/> <span class="error">[38649.247075]</span> LustreError: 12609:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0000_UUID<br/> <span class="error">[38649.247213]</span> LustreError: 12609:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">[38649.247349]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0000_UUID 2:0 3:1 <br/> <span class="error">[38650.948357]</span> Lustre: MGS: Regenerating lus04-OST0003 log by user request.<br/> <span class="error">[38655.146204]</span> LustreError: 12622:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38655.146368]</span> LustreError: 12622:0:(obd_config.c:372:class_setup()) setup lus04-OST0001-osc failed (-2)<br/> <span class="error">[38655.146503]</span> LustreError: 12622:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> <span class="error">[38655.146641]</span> Lustre: cmd=cf003 0:lus04-OST0001-osc 1:lus04-OST0001_UUID 2:0@<0:0> <br/> <span class="error">[38655.453621]</span> Lustre: MGS: Regenerating lus04-OST0004 log by user request.<br/> <span class="error">[38659.318763]</span> LustreError: 12624:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38659.318917]</span> Lustre: 12624:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0001_UUID<br/> <span class="error">[38659.339303]</span> LustreError: 12624:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0001_UUID not set up<br/> <span class="error">[38659.339447]</span> LustreError: 12624:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0001_UUID<br/> <span class="error">[38659.339588]</span> LustreError: 12624:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">[38659.339722]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0001_UUID 2:1 3:1 <br/> <span class="error">[38662.002368]</span> Lustre: MGS: Regenerating lus04-OST0006 log by user request.<br/> <span class="error">[38662.002372]</span> Lustre: Skipped 1 previous similar message<br/> <span class="error">[38666.809608]</span> LustreError: 12633:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38666.809768]</span> LustreError: 12633:0:(obd_config.c:372:class_setup()) setup lus04-OST0002-osc failed (-2)<br/> <span class="error">[38666.809896]</span> LustreError: 12633:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> <span class="error">[38666.810024]</span> Lustre: cmd=cf003 0:lus04-OST0002-osc 1:lus04-OST0002_UUID 2:0@<0:0> <br/> <span class="error">[38672.517992]</span> Lustre: MGS: Regenerating lus04-OST0009 log by user request.<br/> <span class="error">[38672.517996]</span> Lustre: Skipped 2 previous similar messages<br/> <span class="error">[38673.545529]</span> LustreError: 12634:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38673.545678]</span> Lustre: 12634:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0002_UUID<br/> <span class="error">[38673.604532]</span> LustreError: 12635:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1e3:0x0: rc -2<br/> <span class="error">[38673.604684]</span> LustreError: 12635:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1e3:0: rc -2<br/> <span class="error">[38673.604826]</span> LustreError: 12635:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1e3<br/> <span class="error">[38673.605968]</span> LustreError: 12634:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0002_UUID not set up<br/> <span class="error">[38673.606117]</span> LustreError: 12634:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0002_UUID<br/> <span class="error">[38673.606258]</span> LustreError: 12634:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">[38673.606401]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0002_UUID 2:2 3:1 <br/> <span class="error">[38680.560870]</span> LustreError: 12638:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38680.561031]</span> LustreError: 12638:0:(obd_config.c:372:class_setup()) setup lus04-OST0003-osc failed (-2)<br/> <span class="error">[38680.561162]</span> Lustre: cmd=cf003 0:lus04-OST0003-osc 1:lus04-OST0003_UUID 2:0@<0:0> <br/> <span class="error">[38685.703831]</span> LustreError: 12651:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38685.703987]</span> Lustre: 12651:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0003_UUID<br/> <span class="error">[38685.748801]</span> LustreError: 12652:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1e8:0x0: rc -2<br/> <span class="error">[38685.748936]</span> LustreError: 12652:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1e8:0: rc -2<br/> <span class="error">[38685.749062]</span> LustreError: 12652:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1e8<br/> <span class="error">[38685.750193]</span> LustreError: 12651:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0003_UUID not set up<br/> <span class="error">[38685.750328]</span> LustreError: 12651:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0003_UUID<br/> <span class="error">[38685.750468]</span> LustreError: 12651:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">[38685.750608]</span> LustreError: 12651:0:(obd_config.c:1199:class_config_llog_handler()) Skipped 1 previous similar message<br/> <span class="error">[38685.750748]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0003_UUID 2:3 3:1 <br/> <span class="error">[38689.533835]</span> Lustre: MGS: Regenerating lus04-OST000e log by user request.<br/> <span class="error">[38689.533839]</span> Lustre: Skipped 4 previous similar messages<br/> <span class="error">[38693.310511]</span> LustreError: 12659:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38693.310678]</span> LustreError: 12659:0:(obd_config.c:372:class_setup()) setup lus04-OST0004-osc failed (-2)<br/> <span class="error">[38693.310824]</span> Lustre: cmd=cf003 0:lus04-OST0004-osc 1:lus04-OST0004_UUID 2:0@<0:0> <br/> <span class="error">[38724.566711]</span> Lustre: MGS: Regenerating lus04-OST000f log by user request.<br/> <span class="error">[38727.657075]</span> LustreError: 12691:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38727.657250]</span> Lustre: 12691:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0004_UUID<br/> <span class="error">[38727.697633]</span> LustreError: 12692:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1e7:0x0: rc -2<br/> <span class="error">[38727.697806]</span> LustreError: 12692:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1e7:0: rc -2<br/> <span class="error">[38727.697988]</span> LustreError: 12692:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1e7<br/> <span class="error">[38727.699077]</span> LustreError: 12691:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0004_UUID not set up<br/> <span class="error">[38727.699234]</span> LustreError: 12691:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0004_UUID<br/> <span class="error">[38727.699394]</span> LustreError: 12691:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">[38727.699546]</span> LustreError: 12691:0:(obd_config.c:1199:class_config_llog_handler()) Skipped 1 previous similar message<br/> <span class="error">[38727.699700]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0004_UUID 2:4 3:1 <br/> <span class="error">[38739.668093]</span> LustreError: 12693:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38739.668302]</span> LustreError: 12693:0:(obd_config.c:372:class_setup()) setup lus04-OST0005-osc failed (-2)<br/> <span class="error">[38739.668467]</span> Lustre: cmd=cf003 0:lus04-OST0005-osc 1:lus04-OST0005_UUID 2:0@<0:0> <br/> <span class="error">[38745.892767]</span> LustreError: 12705:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38745.892918]</span> Lustre: 12705:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0005_UUID<br/> <span class="error">[38745.936667]</span> LustreError: 12706:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1ec:0x0: rc -2<br/> <span class="error">[38745.936816]</span> LustreError: 12706:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1ec:0: rc -2<br/> <span class="error">[38745.936956]</span> LustreError: 12706:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1ec<br/> <span class="error">[38745.938176]</span> LustreError: 12705:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0005_UUID not set up<br/> <span class="error">[38745.938322]</span> LustreError: 12705:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0005_UUID<br/> <span class="error">[38745.938463]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0005_UUID 2:5 3:1 <br/> <span class="error">[38750.820202]</span> LustreError: 12707:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38750.820379]</span> LustreError: 12707:0:(obd_config.c:372:class_setup()) setup lus04-OST0006-osc failed (-2)<br/> <span class="error">[38750.820521]</span> Lustre: cmd=cf003 0:lus04-OST0006-osc 1:lus04-OST0006_UUID 2:0@<0:0> <br/> <span class="error">[38758.362456]</span> LustreError: 12715:0:(obd_config.c:611:class_add_conn()) try to add conn on immature client dev<br/> <span class="error">[38758.362609]</span> Lustre: 12715:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0006_UUID<br/> <span class="error">[38758.402883]</span> LustreError: 12716:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1eb:0x0: rc -2<br/> <span class="error">[38758.403024]</span> LustreError: 12716:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1eb:0: rc -2<br/> <span class="error">[38758.403159]</span> LustreError: 12716:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1eb<br/> <span class="error">[38758.404281]</span> LustreError: 12715:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0006_UUID not set up<br/> <span class="error">[38758.404420]</span> LustreError: 12715:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0006_UUID<br/> <span class="error">[38758.404559]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0006_UUID 2:6 3:1 <br/> <span class="error">[38762.319547]</span> LustreError: 12718:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> <span class="error">[38762.319692]</span> LustreError: 12718:0:(obd_config.c:1199:class_config_llog_handler()) Skipped 4 previous similar messages<br/> <span class="error">[38762.319835]</span> Lustre: cmd=cf003 0:lus04-OST0007-osc 1:lus04-OST0007_UUID 2:0@<0:0> <br/> <span class="error">[38769.310496]</span> LustreError: 12721:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1f0:0x0: rc -2<br/> <span class="error">[38769.310651]</span> LustreError: 12721:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1f0:0: rc -2<br/> <span class="error">[38769.310794]</span> LustreError: 12721:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1f0<br/> <span class="error">[38769.311941]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0007_UUID 2:7 3:1 <br/> <span class="error">[38775.699590]</span> LustreError: 12735:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> <span class="error">[38775.699736]</span> LustreError: 12735:0:(ldlm_lib.c:333:client_obd_setup()) Skipped 1 previous similar message<br/> <span class="error">[38775.699899]</span> LustreError: 12735:0:(obd_config.c:372:class_setup()) setup lus04-OST0008-osc failed (-2)<br/> <span class="error">[38775.700035]</span> LustreError: 12735:0:(obd_config.c:372:class_setup()) Skipped 1 previous similar message<br/> <span class="error">[38775.700175]</span> Lustre: cmd=cf003 0:lus04-OST0008-osc 1:lus04-OST0008_UUID 2:0@<0:0> </p> <p>Kernel messages from one of the OSS's</p> <p><span class="error">[42070.117243]</span> Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> <span class="error">[42070.215106]</span> Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> <span class="error">[42070.269066]</span> LDISKFS-fs (dm-14): barriers disabled<br/> <span class="error">[42070.290981]</span> LDISKFS-fs (dm-14): mounted filesystem with ordered data mode<br/> <span class="error">[42070.480698]</span> LDISKFS-fs (dm-14): barriers disabled<br/> <span class="error">[42070.490424]</span> LDISKFS-fs (dm-14): mounted filesystem with ordered data mode<br/> <span class="error">[42070.503841]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> <span class="error">[42070.561443]</span> Lustre: Filtering OBD driver; <a href="http://wiki.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/</a><br/> <span class="error">[42070.592412]</span> Lustre: lus04-OST0000: Now serving lus04-OST0000 on /dev/mapper/vd00 with recovery enabled<br/> <span class="error">[42083.497821]</span> LDISKFS-fs (dm-2): barriers disabled<br/> <span class="error">[42083.518965]</span> LDISKFS-fs (dm-2): mounted filesystem with ordered data mode<br/> <span class="error">[42083.681216]</span> LDISKFS-fs (dm-2): barriers disabled<br/> <span class="error">[42083.690062]</span> LDISKFS-fs (dm-2): mounted filesystem with ordered data mode<br/> <span class="error">[42083.972670]</span> Lustre: lus04-OST0002: Now serving lus04-OST0002 on /dev/mapper/vd02 with recovery enabled<br/> <span class="error">[42090.839376]</span> LDISKFS-fs (dm-5): barriers disabled<br/> <span class="error">[42090.859872]</span> LDISKFS-fs (dm-5): mounted filesystem with ordered data mode<br/> <span class="error">[42091.040129]</span> LDISKFS-fs (dm-5): barriers disabled<br/> <span class="error">[42091.048959]</span> LDISKFS-fs (dm-5): mounted filesystem with ordered data mode<br/> <span class="error">[42091.389277]</span> Lustre: lus04-OST0004: Now serving lus04-OST0004 on /dev/mapper/vd04 with recovery enabled<br/> <span class="error">[42097.403696]</span> LDISKFS-fs (dm-13): barriers disabled<br/> <span class="error">[42097.425964]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> <span class="error">[42097.588491]</span> LDISKFS-fs (dm-13): barriers disabled<br/> <span class="error">[42097.597649]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> <span class="error">[42098.006214]</span> Lustre: lus04-OST0006: Now serving lus04-OST0006 on /dev/mapper/vd06 with recovery enabled<br/> <span class="error">[42104.273626]</span> LDISKFS-fs (dm-6): barriers disabled<br/> <span class="error">[42104.295832]</span> LDISKFS-fs (dm-6): mounted filesystem with ordered data mode<br/> <span class="error">[42104.452328]</span> LDISKFS-fs (dm-6): barriers disabled<br/> <span class="error">[42104.461200]</span> LDISKFS-fs (dm-6): mounted filesystem with ordered data mode<br/> <span class="error">[42104.809165]</span> Lustre: lus04-OST0008: Now serving lus04-OST0008 on /dev/mapper/vd08 with recovery enabled<br/> <span class="error">[42111.294964]</span> LDISKFS-fs (dm-1): barriers disabled<br/> <span class="error">[42111.317643]</span> LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> <span class="error">[42111.483790]</span> LDISKFS-fs (dm-1): barriers disabled<br/> <span class="error">[42111.493072]</span> LDISKFS-fs (dm-1): mounted filesystem with ordered data mode<br/> <span class="error">[42111.836007]</span> Lustre: lus04-OST000a: Now serving lus04-OST000a on /dev/mapper/vd10 with recovery enabled<br/> <span class="error">[42117.397322]</span> LDISKFS-fs (dm-3): barriers disabled<br/> <span class="error">[42117.417872]</span> LDISKFS-fs (dm-3): mounted filesystem with ordered data mode<br/> <span class="error">[42117.577000]</span> LDISKFS-fs (dm-3): barriers disabled<br/> <span class="error">[42117.585901]</span> LDISKFS-fs (dm-3): mounted filesystem with ordered data mode<br/> <span class="error">[42124.916269]</span> LDISKFS-fs (dm-12): barriers disabled<br/> <span class="error">[42124.940165]</span> LDISKFS-fs (dm-12): mounted filesystem with ordered data mode<br/> <span class="error">[42125.119575]</span> LDISKFS-fs (dm-12): barriers disabled<br/> <span class="error">[42125.128708]</span> LDISKFS-fs (dm-12): mounted filesystem with ordered data mode<br/> <span class="error">[42125.516470]</span> Lustre: lus04-OST000e: Now serving lus04-OST000e on /dev/mapper/vd14 with recovery enabled<br/> <span class="error">[42125.516474]</span> Lustre: Skipped 1 previous similar message<br/> root@lus04-oss1:~# </p> <p>root@lus04-mds1:~# lctl dl<br/> 0 UP mgs MGS MGS 13<br/> 1 UP mgc MGC172.17.148.4@tcp ec0779df-4709-87c5-fbb3-c1370bccaab5 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 AT osc lus04-OST0000-osc lus04-mdtlov_UUID 1<br/> 6 AT osc lus04-OST0001-osc lus04-mdtlov_UUID 1<br/> 7 AT osc lus04-OST0002-osc lus04-mdtlov_UUID 1<br/> 8 AT osc lus04-OST0003-osc lus04-mdtlov_UUID 1<br/> 9 AT osc lus04-OST0004-osc lus04-mdtlov_UUID 1<br/> 10 AT osc lus04-OST0005-osc lus04-mdtlov_UUID 1<br/> 11 AT osc lus04-OST0006-osc lus04-mdtlov_UUID 1<br/> 12 AT osc lus04-OST0007-osc lus04-mdtlov_UUID 1<br/> 13 AT osc lus04-OST0008-osc lus04-mdtlov_UUID 1</p> <p>"Possibly need a second opinion, but those params don't look at all right to me. I see 2 --mgsnode options + a --failnode option. Think there should be only 1 --mdsnode + ! --failnode.</p> <p>Just for simplification purposes could you try just setting 1 --mgsnode and no --failnode? I understand this probably isn't your desired target config, but in the interest of getting the fs back up to possibly copy files off simpler is better. If we can get that to work we can work on turning failover back on again later."</p> <p>Of course, I also missed the --erase-params on the MGS. Right now we only want to get the file system back for a few days so we can copy the data off and reformat it ( so a day or 2 for 240TB ).</p> <p>After unmounting all the OSS's and attempting to unmount the MDT</p> <p>Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.645245]</span> Lustre: 12744:0:(mds_lov.c:1114:mds_notify()) MDS lus04-MDT0000: add target lus04-OST0008_UUID<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.645247]</span> Lustre: 12744:0:(mds_lov.c:1114:mds_notify()) Skipped 1 previous similar message<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.686343]</span> LustreError: 12745:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x1ef:0x0: rc -2<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.686490]</span> LustreError: 12745:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x1ef:0: rc -2<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.686625]</span> LustreError: 12745:0:(llog_obd.c:291:cat_cancel_cb()) Cannot find handle for log 0x1ef<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.687672]</span> LustreError: 12744:0:(lov_obd.c:289:lov_connect_obd()) Target lus04-OST0008_UUID not set up<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.687816]</span> LustreError: 12744:0:(lov_obd.c:289:lov_connect_obd()) Skipped 1 previous similar message<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.687952]</span> LustreError: 12744:0:(lov_obd.c:727:lov_add_target()) connect or notify failed (-22) for lus04-OST0008_UUID<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.688093]</span> LustreError: 12744:0:(lov_obd.c:727:lov_add_target()) Skipped 1 previous similar message<br/> Oct 18 23:10:38 lus04-mds1 kernel: <span class="error">[38784.688235]</span> Lustre: cmd=cf00d 0:lus04-mdtlov 1:lus04-OST0008_UUID 2:8 3:1 <br/> Oct 18 23:26:35 lus04-mds1 kernel: <span class="error">[39740.309228]</span> Lustre: Failing over lus04-MDT0000<br/> Oct 18 23:26:35 lus04-mds1 kernel: <span class="error">[39740.310959]</span> LustreError: 13468:0:(lov_obd.c:1012:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1<br/> Oct 18 23:27:05 lus04-mds1 kernel: <span class="error">[39770.254590]</span> Lustre: Mount still busy with 14 refs after 30 secs.<br/> Oct 18 23:27:35 lus04-mds1 kernel: <span class="error">[39800.201341]</span> Lustre: Mount still busy with 14 refs after 60 secs.<br/> Oct 18 23:28:05 lus04-mds1 kernel: <span class="error">[39830.148112]</span> Lustre: Mount still busy with 14 refs after 90 secs.<br/> Oct 18 23:28:35 lus04-mds1 kernel: <span class="error">[39860.094852]</span> Lustre: Mount still busy with 14 refs after 120 secs.<br/> Oct 18 23:29:05 lus04-mds1 kernel: <span class="error">[39890.041601]</span> Lustre: Mount still busy with 14 refs after 150 secs.<br/> Oct 18 23:29:35 lus04-mds1 kernel: <span class="error">[39919.988359]</span> Lustre: Mount still busy with 14 refs after 180 secs.<br/> Oct 18 23:30:05 lus04-mds1 kernel: <span class="error">[39949.935117]</span> Lustre: Mount still busy with 14 refs after 210 secs.<br/> Oct 18 23:30:35 lus04-mds1 kernel: <span class="error">[39979.881872]</span> Lustre: Mount still busy with 14 refs after 240 secs.</p> <p>How should I proceed on the unmount of the MDT</p> <p>Try umount -f on the MDT. If that doesn't work then just force reboot the server & bring it up again.</p> <p>Looking back over the history here I suspect something is wrong with the failover config.</p> <p>All the tunefs params show up with no_primnode flags set. Also show a failnode param set. Some of the earlier logs suggest that MGS & MDT is attempting to communicate with the failover nid when you are mounting OSTs on the primary server. Think we may get a bit farther without failover for now.</p> <p>I am going to be doing that in an instant, but I thought it would be worth mentioning that this filesystem was originally formatted 3+years ago with both possible servers specified as failover nids, not just the alternative host.</p> <p>This is from my notes from, note that we had two failnodes configured one of which was the "primary" node.</p> <p><span class="error">[15/10/2013 12:22:35]</span> James Beal: <br/> root@lus04-oss1:~# tunefs.lustre --erase-params --writeconf --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.6@tcp --failnode 172.17.148.7@tcp --param ost.quota_type=ug /dev/mapper/vd08<br/> checking for existing Lustre data: found<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-OST0008<br/> Index: 8<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x2<br/> (OST )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.5@tcp mgsnode=172.17.148.4@tcp failover.node=172.17.148.6@tcp failover.node=172.17.148.7@tcp ost.quota_type=ug2</p> <p> Permanent disk data:<br/> Target: lus04=OST0008<br/> Index: 8<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x142<br/> (OST update writeconf )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.6@tcp failover.node=172.17.148.7@tcp ost.quota_type=ug</p> <p>Writing CONFIGS/mountdata</p> <p>hmm. I notice that there is never a --fsname= option in your tunefs cmds, but the LustreFS reported in the params being set is still lus04. Suggests to me that there is only some partial erasure going on.</p> <p>Just to satisfy me could you do a round of tunefs where the only cmd line option is --erase-params then follow that with another round where you do --writeconf with all the desired params, including --fsname=lus04, but no --erase-params?</p> <p>It occurs to me to ask are you possibly running the same tunefs cmds for a given device on both the primary node and the backup node where that device is visible? Think that would have very ill effects.</p> <p>"It occurs to me to ask are you possibly running the same tunefs cmds for a given device on both the primary node and the backup node where that device is visible? Think that would have very ill effects."</p> <p>It is always worth asking those questions but no, I am only running the tunefs on the active node, the MGS and MDT I do by hand and the OSS I am keying off the commented mounts in /etc/fstab. I am just about to do this with the OSS(s) now.</p> <p>root@lus04-mds1:~# tunefs.lustre --erase-params /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata<br/> root@lus04-mds1:~# tunefs.lustre --erase-params /dev/lus04-mdt0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1001<br/> (MDT no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1041<br/> (MDT update no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata<br/> root@lus04-mds1:~# tunefs.lustre --writeconf --mgs --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata<br/> root@lus04-mds1:~# tunefs.lustre --writeconf --mgsnode 172.17.148.4@tcp --fsname=lus04 /dev/lus04-mdt0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1041<br/> (MDT update no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1141<br/> (MDT update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p>Writing CONFIGS/mountdata<br/> root@lus04-mds1:~# </p> <p>And a sample OSS</p> <p>tunefs.lustre --erase-params </p> <p>Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters:</p> <p>Writing CONFIGS/mountdata</p> <p>tunefs.lustre --ost --writeconf --mgsnode 172.17.148.4@tcp --fsname=lus04 </p> <p>Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>I havn't attempted to mount them yet as the unmounting of the MDT does not go well.</p> <p>I see all your nodes still have no_primnode flag. Have been trying to get that set on a local test fs, can't figure out how. Looking at the code of mkfs.lustre it seems to be associated with setting a failover, failnode, or servicenode option. But none of those options are in your recent tunefs commands. Seems like some things are not getting cleared by erase or writeconf. I may need to consult wiser experts.</p> <p>In the last few days this file system has had tunefs.lustre run a lot, we have used servicenode, failnode and had it having failover or no failover. I do have a record of what has happened with the file system but it is over 100 pages in word....</p> <p>It is now 12:45 in the morning local time for me, I appreciate your efforts. If there is anything more I can do then add it to the ticket and I will try and do it tomorrow family permitting. As I said this system was formatted with say on .4, failnode=.4, failnode=.5 which worked well enough however it feels that fixing that broke things but I have no evidence for that.</p> <p>Sorry, I didn't realize it was so late for you. by all means quit for now. I will try to check back on this ticket over the weekend, but can't promise. Thanks for your quick response and turnaround on my requests.</p> <p>"Sorry, I didn't realize it was so late for you. by all means quit for now. I will try to check back on this ticket over the weekend, but can't promise. Thanks for your quick response and turnaround on my requests."</p> <p>I realise that with you in the states that we will not have that much overlap so I am happy to stay up late to get the support. I completely understand about the weekend. Have a good weekend and if anything comes up then put it here and I will attend to it as promptly as I can manage <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Part one of try everything on the failnode and see if it all works if a prime node is not defined.</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params --mgs --failnode 172.17.148.5@tcp --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: failover.node=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: failover.node=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mgsnode=172.17.148.5@tcp --failnode 172.17.148.5@tcp --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: failover.node=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>Now to mount these on their partner node .5</p> <p>Oct 19 13:22:51 lus04-mds2 kernel: <span class="error">[94212.007840]</span> Lustre: MGS MGS started<br/> Oct 19 13:22:51 lus04-mds2 kernel: <span class="error">[94212.008444]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.421033]</span> LDISKFS-fs (dm-0): barriers disabled<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.424375]</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.538080]</span> LDISKFS-fs (dm-0): barriers disabled<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.540969]</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.557986]</span> Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.574965]</span> Lustre: Enabling user_xattr<br/> Oct 19 13:23:05 lus04-mds2 kernel: <span class="error">[94225.604777]</span> Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/mapper/lus04--mdt0-lus04 with recovery enabled</p> <p>tunefs.lustre --ost --writeconf --mgsnode 172.17.148.5@tcp --fsname=lus04 --ost --failnode=172.17.148.7@tcp /dev/mapper/vd00</p> <p>And to mount on the partner pair, note " sent from MGC172.17.148.4@tcp to NID 172.17.148.4@tcp 105s ago has timed out", The MGS is mounted on .5</p> <p>Oct 19 13:27:55 lus04-oss2 kernel: <span class="error">[93559.887640]</span> LDISKFS-fs (dm-13): barriers disabled<br/> Oct 19 13:27:55 lus04-oss2 kernel: <span class="error">[93559.910142]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> Oct 19 13:27:55 lus04-oss2 kernel: <span class="error">[93560.104185]</span> LDISKFS-fs (dm-13): barriers disabled<br/> Oct 19 13:27:55 lus04-oss2 kernel: <span class="error">[93560.113434]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> Oct 19 13:29:40 lus04-oss2 kernel: <span class="error">[93664.929907]</span> Lustre: 32416:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1449276099002414 sent from MGC172.17.148.4@tcp to NID 172.17.148.4@tcp 105s ago has timed out (105s prior to deadline).<br/> Oct 19 13:29:40 lus04-oss2 kernel: <span class="error">[93664.929910]</span> req@ffff8802f5e27c00 x1449276099002414/t0 o250->MGS@MGC172.17.148.4@tcp_0:26/25 lens 368/584 e 0 to 1 dl 1382185780 ref 1 fl Rpc:N/0/0 rc 0/0<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.895825]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.914641]</span> LustreError: 2228:0:(llog.c:381:llog_process()) cannot start thread: -513<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.914671]</span> LustreError: 2228:0:(mgc_request.c:1094:mgc_copy_llog()) Failed to copy remote log lus04-OST0000 (-513)<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944136]</span> LustreError: 2228:0:(llog.c:381:llog_process()) cannot start thread: -513<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944173]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-OST0000' failed (-513). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944224]</span> LustreError: 2228:0:(obd_mount.c:1143:server_start_targets()) failed to start server lus04-OST0000: -513<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944253]</span> LustreError: 2228:0:(obd_mount.c:1672:server_fill_super()) Unable to start targets: -513<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944381]</span> LustreError: 2228:0:(obd_mount.c:1455:server_put_super()) no obd lus04-OST0000<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944531]</span> LustreError: 2228:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93684.944662]</span> LustreError: 2228:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.048913]</span> Lustre: server umount lus04-OST0000 complete<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.048919]</span> LustreError: 2228:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-513)<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.076179]</span> LDISKFS-fs (dm-13): barriers disabled<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.085465]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.194214]</span> LDISKFS-fs (dm-13): barriers disabled<br/> Oct 19 13:30:00 lus04-oss2 kernel: <span class="error">[93685.203402]</span> LDISKFS-fs (dm-13): mounted filesystem with ordered data mode<br/> Oct 19 13:30:04 lus04-oss2 kernel: <span class="error">[93689.375697]</span> LustreError: 137-5: UUID 'lus04-OST0000_UUID' is not available for connect (no target)<br/> Oct 19 13:30:04 lus04-oss2 kernel: <span class="error">[93689.375831]</span> LustreError: 2735:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (<del>19) req@ffff880614092000 x1449326212546592/t0 o8</del>><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1382185904 ref 1 fl Interpret:/0/0 rc -19/0<br/> Oct 19 13:31:45 lus04-oss2 kernel: <span class="error">[93790.023923]</span> Lustre: 32416:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1449276099002422 sent from MGC172.17.148.4@tcp to NID 172.17.148.4@tcp 105s ago has timed out (105s prior to deadline).<br/> Oct 19 13:31:45 lus04-oss2 kernel: <span class="error">[93790.023926]</span> req@ffff8802f5a62800 x1449276099002422/t0 o250->MGS@MGC172.17.148.4@tcp_0:26/25 lens 368/584 e 0 to 1 dl 1382185905 ref 1 fl Rpc:N/0/0 rc 0/0<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.663025]</span> LustreError: 2228:0:(mgc_request.c:365:mgc_requeue_add()) log lus04-OST0000: cannot start requeue thread (-513),no more log updates!<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.663175]</span> LustreError: 2228:0:(mgc_request.c:638:mgc_blocking_ast()) cancel CB failed -513:<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.663308]</span> LustreError: 2228:0:(mgc_request.c:639:mgc_blocking_ast()) ### MGC ast ns: MGC172.17.148.4@tcp lock: ffff8802a8324e00/0x85ea67077e7a36d8 lrc: 5/0,0 mode: --/CR res: 224151172460/0 rrc: 1 type: PLN flags: 0x4002c90 remote: 0x0 expref: -99 pid: 2228 timeout 0<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.664025]</span> LustreError: 2228:0:(llog.c:381:llog_process()) cannot start thread: -513<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.664164]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-OST0000' failed (-513). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.664426]</span> LustreError: 2228:0:(obd_mount.c:1143:server_start_targets()) failed to start server lus04-OST0000: -513<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.664560]</span> LustreError: 2228:0:(obd_mount.c:1672:server_fill_super()) Unable to start targets: -513<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.664741]</span> LustreError: 2228:0:(obd_mount.c:1455:server_put_super()) no obd lus04-OST0000<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.801800]</span> Lustre: server umount lus04-OST0000 complete<br/> Oct 19 13:31:55 lus04-oss2 kernel: <span class="error">[93799.801806]</span> LustreError: 2228:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-513)</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params --mgs --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata</p> <p>At this point I made a mistake, I wrote the MDT config into the MGS by accident.</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params --mgsnode=172.17.148.4@tcp --mdt --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1144<br/> (MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: lus04-MDTffff<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1145<br/> (MDT MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>I tried to reverse this.</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1145<br/> (MDT MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1145<br/> (MDT MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata</p> <p>I now have two mistakes I need to fix in the flags.</p> <p>root@lus04-mds1:~# tunefs.lustre --writeconf --erase-params --mgs --fsname=lus04 /dev/lus04-mgs0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1145<br/> (MDT MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1145<br/> (MDT MGS update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>Writing CONFIGS/mountdata</p> <p>I will copy a backup I did of the MGS ( on thursday ) over to the machine locally and attempt to mount that as a loop device.</p> <p>I can now get the OSS to mount and it looks ok however a client will not mount</p> <p>Start with a clean MGS</p> <p>root@lus04-mds1:/# dd if=/dev/zero bs=1M count=128 of=/home/MGS_DONOT_DELETE<br/> 128+0 records in<br/> 128+0 records out<br/> 134217728 bytes (134 MB) copied, 0.0937643 s, 1.4 GB/s<br/> root@lus04-mds1:/# losetup /dev/loop2 /home/MGS_DONOT_DELETE<br/> root@lus04-mds1:/# mkfs.lustre --mgs --fsname=lus04 /dev/loop2</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p>checking for existing Lustre data: not found<br/> device size = 128MB<br/> formatting backing filesystem ldiskfs on /dev/loop2<br/> target name MGS<br/> 4k blocks 32768<br/> options -q -O uninit_bg,dir_nlink -E lazy_journal_init -F<br/> mkfs_cmd = mke2fs -j -b 4096 -L MGS -q -O uninit_bg,dir_nlink -E lazy_journal_init -F /dev/loop2 32768<br/> Writing CONFIGS/mountdata</p> <p>root@lus04-mds1:~# mount -t lustre /dev/loop2 /export/MGS</p> <p>And the MDT</p> <p>tunefs.lustre /dev/lus04-mdt0/lus04 <br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1001<br/> (MDT no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1001<br/> (MDT no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p>exiting before disk write.</p> <p>Do all the OSS mounts on the partners and look at the devices</p> <p>root@lus04-mds1:/root# lctl dl<br/> 0 UP mgs MGS MGS 5<br/> 1 UP mgc MGC172.17.148.4@tcp 5024ddc3-8729-483f-148c-5bbfe6326be2 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 UP osc lus04-OST0000-osc lus04-mdtlov_UUID 5<br/> 6 UP osc lus04-OST0001-osc lus04-mdtlov_UUID 5<br/> 7 UP osc lus04-OST0002-osc lus04-mdtlov_UUID 5<br/> 8 UP osc lus04-OST0003-osc lus04-mdtlov_UUID 5<br/> 9 UP osc lus04-OST0004-osc lus04-mdtlov_UUID 5<br/> 10 UP osc lus04-OST0005-osc lus04-mdtlov_UUID 5<br/> 11 UP osc lus04-OST0006-osc lus04-mdtlov_UUID 5<br/> 12 UP osc lus04-OST0007-osc lus04-mdtlov_UUID 5<br/> 13 UP osc lus04-OST0008-osc lus04-mdtlov_UUID 5<br/> 14 UP osc lus04-OST0009-osc lus04-mdtlov_UUID 5<br/> 15 UP osc lus04-OST000a-osc lus04-mdtlov_UUID 5<br/> 16 UP osc lus04-OST000b-osc lus04-mdtlov_UUID 5<br/> 17 UP osc lus04-OST000c-osc lus04-mdtlov_UUID 5<br/> 18 UP osc lus04-OST000d-osc lus04-mdtlov_UUID 5<br/> 19 UP osc lus04-OST000e-osc lus04-mdtlov_UUID 5<br/> 20 UP osc lus04-OST000f-osc lus04-mdtlov_UUID 5<br/> 21 UP osc lus04-OST0010-osc lus04-mdtlov_UUID 5<br/> 22 UP osc lus04-OST0011-osc lus04-mdtlov_UUID 5<br/> 23 UP osc lus04-OST0012-osc lus04-mdtlov_UUID 5<br/> 24 UP osc lus04-OST0013-osc lus04-mdtlov_UUID 5<br/> 25 UP osc lus04-OST0014-osc lus04-mdtlov_UUID 5<br/> 26 UP osc lus04-OST0015-osc lus04-mdtlov_UUID 5<br/> 27 UP osc lus04-OST0016-osc lus04-mdtlov_UUID 5<br/> 28 UP osc lus04-OST0017-osc lus04-mdtlov_UUID 5<br/> 29 UP osc lus04-OST0018-osc lus04-mdtlov_UUID 5<br/> 30 UP osc lus04-OST0019-osc lus04-mdtlov_UUID 5<br/> 31 UP osc lus04-OST001a-osc lus04-mdtlov_UUID 5<br/> 32 UP osc lus04-OST001b-osc lus04-mdtlov_UUID 5<br/> 33 UP osc lus04-OST001c-osc lus04-mdtlov_UUID 5<br/> 34 UP osc lus04-OST001d-osc lus04-mdtlov_UUID 5</p> <p>See that a client mount fails</p> <p>mount /lustre/scratch104<br/> mount.lustre: mount lus04-mds1@tcp0:/lus04 at /lustre/scratch104 failed: No such file or directory<br/> Is the MGS specification correct?<br/> Is the filesystem name correct?<br/> If upgrading, is the copied client log valid? (see upgrade docs)<br/> root@bc-11-2-07:~# tail -f /var/log/kern.log<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.814834]</span> Lustre: cmd=cf003 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.815018]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.815345]</span> LustreError: 11016:0:(llite_lib.c:1099:ll_fill_super()) Unable to process log: -2<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.815751]</span> LustreError: 11016:0:(obd_config.c:443:class_cleanup()) Device 68 not setup<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.816019]</span> LustreError: 11016:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.816192]</span> LustreError: 11016:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 1 previous similar message<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.816344]</span> LustreError: 11016:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.816497]</span> LustreError: 11016:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) Skipped 1 previous similar message<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.817443]</span> Lustre: client lus04-client(ffff88015c402800) umount complete<br/> Oct 19 16:26:29 bc-11-2-07 kernel: <span class="error">[17774424.817530]</span> LustreError: 11016:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-2)</p> <p>Clean out the client config from the MGS</p> <p>root@lus04-mds1:/# umount /export/MGS<br/> root@lus04-mds1:/# mount -t ldiskfs /dev/loop1 /export/MGS<br/> root@lus04-mds1:/# cd /export/MGS/CONFIGS<br/> root@lus04-mds1:/export/MGS/CONFIGS# mkdir ../CONFIGS_OLD1<br/> root@lus04-mds1:/export/MGS/CONFIGS# mv lus04-client ../CONFIGS_OLD1<br/> root@lus04-mds1:/export/MGS/CONFIGS# cd /<br/> root@lus04-mds1:/# umount /export/MGS</p> <p>Now tunefs the OSS so that they believe they are owned by one server, but have a failover mid</p> <p>root@lus04-oss1:~# for i in `seq -w 00 07 ` ; do tunefs.lustre --ost --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 --ost --failnode=172.17.148.7@tcp /dev/mapper/vd$i ; done</p> <p>Now mount them on the partner node</p> <p>root@lus04-oss2:~# for i in `seq -w 00 07 ` ; do mount -t lustre /dev/mapper/vd$i /export/vd$i; done</p> <p>See that the MGS/MDT looks ok</p> <p>root@lus04-mds1:~# lctl dl<br/> 0 UP mgs MGS MGS 13<br/> 1 UP mgc MGC172.17.148.4@tcp 75489c46-b5a0-50ff-86e9-3f688e8a1de8 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 UP osc lus04-OST0000-osc lus04-mdtlov_UUID 5<br/> 6 UP osc lus04-OST0001-osc lus04-mdtlov_UUID 5<br/> 7 UP osc lus04-OST0002-osc lus04-mdtlov_UUID 5<br/> 8 UP osc lus04-OST0003-osc lus04-mdtlov_UUID 5<br/> 9 UP osc lus04-OST0004-osc lus04-mdtlov_UUID 5<br/> 10 UP osc lus04-OST0005-osc lus04-mdtlov_UUID 5<br/> 11 UP osc lus04-OST0006-osc lus04-mdtlov_UUID 5<br/> 12 UP osc lus04-OST0007-osc lus04-mdtlov_UUID 5<br/> 13 UP osc lus04-OST0008-osc lus04-mdtlov_UUID 5<br/> 14 UP osc lus04-OST0009-osc lus04-mdtlov_UUID 5<br/> 15 UP osc lus04-OST000a-osc lus04-mdtlov_UUID 5<br/> 16 UP osc lus04-OST000b-osc lus04-mdtlov_UUID 5<br/> 17 UP osc lus04-OST000c-osc lus04-mdtlov_UUID 5<br/> 18 UP osc lus04-OST000d-osc lus04-mdtlov_UUID 5<br/> 19 UP osc lus04-OST000e-osc lus04-mdtlov_UUID 5<br/> 20 UP osc lus04-OST000f-osc lus04-mdtlov_UUID 5<br/> 21 UP osc lus04-OST0010-osc lus04-mdtlov_UUID 5<br/> 22 UP osc lus04-OST0011-osc lus04-mdtlov_UUID 5<br/> 23 UP osc lus04-OST0012-osc lus04-mdtlov_UUID 5<br/> 24 UP osc lus04-OST0013-osc lus04-mdtlov_UUID 5<br/> 25 UP osc lus04-OST0014-osc lus04-mdtlov_UUID 5<br/> 26 UP osc lus04-OST0015-osc lus04-mdtlov_UUID 5<br/> 27 UP osc lus04-OST0016-osc lus04-mdtlov_UUID 5<br/> 28 UP osc lus04-OST0017-osc lus04-mdtlov_UUID 5<br/> 29 UP osc lus04-OST0018-osc lus04-mdtlov_UUID 5<br/> 30 UP osc lus04-OST0019-osc lus04-mdtlov_UUID 5<br/> 31 UP osc lus04-OST001a-osc lus04-mdtlov_UUID 5<br/> 32 UP osc lus04-OST001b-osc lus04-mdtlov_UUID 5<br/> 33 UP osc lus04-OST001c-osc lus04-mdtlov_UUID 5<br/> 34 UP osc lus04-OST001d-osc lus04-mdtlov_UUID 5<br/> root@lus04-mds1:~# </p> <p>Try and mount the client and get frustrated.<br/> Oct 19 16:49:04 bc-11-2-07 kernel: <span class="error">[17775779.740161]</span> Lustre: Removed LNI 172.17.115.32@tcp<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.590501]</span> Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.662391]</span> Lustre: Added LNI 172.17.115.32@tcp <span class="error">[8/256/0/180]</span><br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.662573]</span> Lustre: Accept secure, port 988<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.752886]</span> Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.773139]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.775766]</span> LustreError: 11961:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.775996]</span> LustreError: 11961:0:(obd_config.c:372:class_setup()) setup lus04-MDT0000-mdc-ffff88041200d000 failed (-2)<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.776149]</span> LustreError: 11961:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.776301]</span> Lustre: cmd=cf003 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.776486]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.776819]</span> LustreError: 11891:0:(llite_lib.c:1099:ll_fill_super()) Unable to process log: -2<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.777235]</span> LustreError: 11891:0:(obd_config.c:443:class_cleanup()) Device 2 not setup<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.777504]</span> LustreError: 11891:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.777659]</span> LustreError: 11891:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.786546]</span> Lustre: client lus04-client(ffff88041200d000) umount complete<br/> Oct 19 16:49:12 bc-11-2-07 kernel: <span class="error">[17775787.786635]</span> LustreError: 11891:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-2)</p> <p>Try and reset the lus04-client log</p> <p>root@lus04-mds1:~# umount /export/MGS<br/> root@lus04-mds1:~# mount -t ldiskfs /dev/loop2 /export/MGS<br/> root@lus04-mds1:~# cd /export/MGS/CONFIGS/<br/> root@lus04-mds1:/export/MGS/CONFIGS# mkdir ../CONFIGS1<br/> root@lus04-mds1:/export/MGS/CONFIGS# mv lus04-client ../CONFIGS1<br/> root@lus04-mds1:/export/MGS/CONFIGS# cd /<br/> root@lus04-mds1:/# umount /export/MGS<br/> root@lus04-mds1:/# mount -t lustre /dev/loop2 /export/MGS</p> <p>Mount a client</p> <p>Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.418669]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.421159]</span> LustreError: 156-2: The client profile 'lus04-client' could not be read from the MGS. Does that filesystem exist?<br/> Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.421516]</span> LustreError: 11971:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.421688]</span> LustreError: 11971:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.422630]</span> Lustre: client lus04-client(ffff8801c6b9d400) umount complete<br/> Oct 19 16:51:43 bc-11-2-07 kernel: <span class="error">[17775938.422713]</span> LustreError: 11971:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-22)</p> <p>Reintroduce the MDT </p> <p>root@lus04-mds1:/# umount /export/MDT0<br/> root@lus04-mds1:/# mount -t lustre /dev/lus04-mdt0/lus04 /export/MDT0</p> <p>And try the client mount again.</p> <p>Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.316139]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.319088]</span> LustreError: 11998:0:(ldlm_lib.c:333:client_obd_setup()) can't add initial connection<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.319323]</span> LustreError: 11998:0:(obd_config.c:372:class_setup()) setup lus04-MDT0000-mdc-ffff88040f93fc00 failed (-2)<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.319477]</span> LustreError: 11998:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.319628]</span> Lustre: cmd=cf003 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.319815]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.320147]</span> LustreError: 11988:0:(llite_lib.c:1099:ll_fill_super()) Unable to process log: -2<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.320558]</span> LustreError: 11988:0:(obd_config.c:443:class_cleanup()) Device 2 not setup<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.320812]</span> LustreError: 11988:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 19 16:52:23 bc-11-2-07 kernel: <span class="error">[17775979.320983]</span> LustreError: 11988:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 19 16:52:24 bc-11-2-07 kernel: <span class="error">[17775979.329645]</span> Lustre: client lus04-client(ffff88040f93fc00) umount complete<br/> Oct 19 16:52:24 bc-11-2-07 kernel: <span class="error">[17775979.329734]</span> LustreError: 11988:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-2)</p> <p>losetup /dev/loop3 /nfs/ssg_data01/jb23/lus04-mdt_in_use_talk_to_me_first <br/> tune2fs -O ^quota /dev/loop3 <br/> tune2fs 1.42.7.wc1 (12-Apr-2013)</p> <p>tunefs.lustre --dryrun /dev/loop3<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04=MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1101<br/> (MDT writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1101<br/> (MDT writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.4@tcp</p> <p>exiting before disk write.</p> <p>tunefs.lustre --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 /dev/loop3<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04=MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1101<br/> (MDT writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1141<br/> (MDT update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>Unmount the OSS's, and reset the config,.</p> <p>root@lus04-oss1:~# for i in `seq -w 08 14` ; do tunefs.lustre --ost --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 --ost --failnode=172.17.148.7@tcp /dev/mapper/vd$i ; done<br/> root@lus04-oss2:~# for i in `seq -w 00 07 ` ; do tunefs.lustre --ost --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 --ost --failnode=172.17.148.6@tcp /dev/mapper/vd$i ; done<br/> root@lus04-oss3:~# for i in `seq -w 15 22` ; do tunefs.lustre --ost --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 --ost --failnode=172.17.148.9@tcp /dev/mapper/vd$i ; done<br/> root@lus04-oss4:~# for i in `seq -w 23 29` ; do tunefs.lustre --ost --writeconf --erase-params --mgsnode 172.17.148.4@tcp --fsname=lus04 --ost --failnode=172.17.148.8@tcp /dev/mapper/vd$i ; done</p> <p>root@lus04-mds1:~# mount | grep lus<br/> /dev/loop2 on /export/MGS type lustre (rw)<br/> root@lus04-mds1:~# mount -t lustre /dev/loop3 /export/MDT0<br/> root@lus04-mds1:~# tail /var/log/kern.log<br/> Oct 20 08:09:44 lus04-mds1 kernel: <span class="error">[66192.187279]</span> LDISKFS-fs (loop3): barriers disabled<br/> Oct 20 08:09:44 lus04-mds1 kernel: <span class="error">[66192.214036]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.710174]</span> LDISKFS-fs (loop3): barriers disabled<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.712679]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.790005]</span> LDISKFS-fs (loop3): barriers disabled<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.791717]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.803940]</span> Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.806465]</span> Lustre: Enabling user_xattr<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.889602]</span> Lustre: Mounting lus04-MDT0000 at first time on 2.0 FS, remove all clients for interop needs<br/> Oct 20 08:10:25 lus04-mds1 kernel: <span class="error">[66232.941098]</span> Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/loop3 with recovery enabled</p> <p>root@lus04-oss1:~# for i in `seq -w 00 07` ; do mount -t lustre /dev/mapper/vd$i /export/vd$i ; done<br/> root@lus04-oss2:~# for i in `seq -w 08 14` ; do mount -t lustre /dev/mapper/vd$i /export/vd$i ; done<br/> root@lus04-oss4:~# for i in `seq -w 15 22` ; do mount -t lustre /dev/mapper/vd$i /export/vd$i ; done<br/> root@lus04-oss3:~# for i in `seq -w 23 29` ; do mount -t lustre /dev/mapper/vd$i /export/vd$i ; done</p> <p>Wait for a bit<br/> grep -i statu /proc/fs/lustre/obdfilter/lus04-OST*/recovery_status<br/> /proc/fs/lustre/obdfilter/lus04-OST0017/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST0018/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST0019/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST001a/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST001b/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST001c/recovery_status:status: RECOVERING<br/> /proc/fs/lustre/obdfilter/lus04-OST001d/recovery_status:status: RECOVERING</p> <p>root@lus04-mds1:~# lctl dl<br/> 0 UP mgs MGS MGS 13<br/> 1 UP mgc MGC172.17.148.4@tcp f450e477-6ad3-c2d7-72f2-ace4ad6d6513 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 UP osc lus04-OST0000-osc lus04-mdtlov_UUID 5<br/> 6 UP osc lus04-OST0001-osc lus04-mdtlov_UUID 5<br/> 7 UP osc lus04-OST0002-osc lus04-mdtlov_UUID 5<br/> 8 UP osc lus04-OST0003-osc lus04-mdtlov_UUID 5<br/> 9 UP osc lus04-OST0004-osc lus04-mdtlov_UUID 5<br/> 10 UP osc lus04-OST0005-osc lus04-mdtlov_UUID 5<br/> 11 UP osc lus04-OST0006-osc lus04-mdtlov_UUID 5<br/> 12 UP osc lus04-OST0007-osc lus04-mdtlov_UUID 5<br/> 13 UP osc lus04-OST0008-osc lus04-mdtlov_UUID 5<br/> 14 UP osc lus04-OST0009-osc lus04-mdtlov_UUID 5<br/> 15 UP osc lus04-OST000a-osc lus04-mdtlov_UUID 5<br/> 16 UP osc lus04-OST000b-osc lus04-mdtlov_UUID 5<br/> 17 UP osc lus04-OST000c-osc lus04-mdtlov_UUID 5<br/> 18 UP osc lus04-OST000d-osc lus04-mdtlov_UUID 5<br/> 19 UP osc lus04-OST000e-osc lus04-mdtlov_UUID 5<br/> 20 UP osc lus04-OST000f-osc lus04-mdtlov_UUID 5<br/> 21 UP osc lus04-OST0010-osc lus04-mdtlov_UUID 5<br/> 22 UP osc lus04-OST0011-osc lus04-mdtlov_UUID 5<br/> 23 UP osc lus04-OST0012-osc lus04-mdtlov_UUID 5<br/> 24 UP osc lus04-OST0013-osc lus04-mdtlov_UUID 5<br/> 25 UP osc lus04-OST0014-osc lus04-mdtlov_UUID 5<br/> 26 UP osc lus04-OST0015-osc lus04-mdtlov_UUID 5<br/> 27 UP osc lus04-OST0016-osc lus04-mdtlov_UUID 5<br/> 28 UP osc lus04-OST0017-osc lus04-mdtlov_UUID 5<br/> 29 UP osc lus04-OST0018-osc lus04-mdtlov_UUID 5<br/> 30 UP osc lus04-OST0019-osc lus04-mdtlov_UUID 5<br/> 31 UP osc lus04-OST001a-osc lus04-mdtlov_UUID 5<br/> 32 UP osc lus04-OST001b-osc lus04-mdtlov_UUID 5<br/> 33 UP osc lus04-OST001c-osc lus04-mdtlov_UUID 5<br/> 34 UP osc lus04-OST001d-osc lus04-mdtlov_UUID 5<br/> root@lus04-mds1:~# </p> <p>Mounting client fails again.</p> <p>Try and reset the client log</p> <p>root@lus04-mds1:~# losetup -a<br/> /dev/loop0: <span class="error">[001c]</span>:5830 (/mnt/users/jb23/lus04-mgs_in_use)<br/> /dev/loop1: <span class="error">[001c]</span>:5830 (/mnt/users/jb23/lus04-mgs_in_use)<br/> /dev/loop2: <span class="error">[0846]</span>:134525730 (/home/MGS_DONOT_DELETE)<br/> /dev/loop3: <span class="error">[0019]</span>:10440828 (/nfs/ssg_data01/jb23/lus04-mdt_in_use_talk_to_me_first)<br/> root@lus04-mds1:~# mount -t ldiskfs /dev/loop3 /export/MDT0<br/> root@lus04-mds1:~# mount -t ldiskfs /dev/loop2 /export/MGS<br/> root@lus04-mds1:~# cd /export/MDT0/<br/> CATALOGS lost+found/ oi.16.10 oi.16.20 oi.16.30 oi.16.40 oi.16.50 oi.16.60 REMOTE_PARENT_DIR/<br/> changelog_catalog lov_objid oi.16.11 oi.16.21 oi.16.31 oi.16.41 oi.16.51 oi.16.61 ROOT/<br/> changelog_users lov_objseq oi.16.12 oi.16.22 oi.16.32 oi.16.42 oi.16.52 oi.16.62 seq_ctl<br/> CONFIGS/ lquota.group oi.16.13 oi.16.23 oi.16.33 oi.16.43 oi.16.53 oi.16.63 seq_srv<br/> CONFIGS_OLD/ lquota.user oi.16.14 oi.16.24 oi.16.34 oi.16.44 oi.16.54 oi.16.7 <br/> fld lquota_v2.group oi.16.15 oi.16.25 oi.16.35 oi.16.45 oi.16.55 oi.16.8 <br/> health_check lquota_v2.user oi.16.16 oi.16.26 oi.16.36 oi.16.46 oi.16.56 oi.16.9 <br/> last_rcvd O/ oi.16.17 oi.16.27 oi.16.37 oi.16.47 oi.16.57 OI_scrub <br/> lfsck_bookmark OBJECTS/ oi.16.18 oi.16.28 oi.16.38 oi.16.48 oi.16.58 PENDING/ <br/> lfsck_namespace oi.16.0 oi.16.19 oi.16.29 oi.16.39 oi.16.49 oi.16.59 quota_master/ <br/> LOGS/ oi.16.1 oi.16.2 oi.16.3 oi.16.4 oi.16.5 oi.16.6 quota_slave/ <br/> root@lus04-mds1:~# cd /export/MDT0/CONFIGS<br/> root@lus04-mds1:/export/MDT0/CONFIGS# mkdir ../C<br/> CATALOGS CONFIGS/ CONFIGS_OLD/ <br/> root@lus04-mds1:/export/MDT0/CONFIGS# mkdir ../CONFIGS_OLD1<br/> root@lus04-mds1:/export/MDT0/CONFIGS# mv lus04-client ../CONFIGS_OLD1<br/> root@lus04-mds1:/export/MDT0/CONFIGS# cd /<br/> root@lus04-mds1:/# umount /export/MDT0<br/> root@lus04-mds1:/# cd /export/MGS/CONFIGS/<br/> root@lus04-mds1:/export/MGS/CONFIGS# mkdir ../CONFIGS1<br/> root@lus04-mds1:/export/MGS/CONFIGS# mv lus04-client ../CONFIGS1<br/> root@lus04-mds1:/export/MGS/CONFIGS# cd /<br/> root@lus04-mds1:/# umount /export/MGS<br/> root@lus04-mds1:/# lustre_rmmod <br/> fsfilt_ldiskfs 75227 0 <br/> obdclass 582304 10 mds,fsfilt_ldiskfs,mgs,mgc,lustre,lov,mdc,lquota,osc,ptlrpc<br/> lvfs 43190 12 mds,fsfilt_ldiskfs,mgs,mgc,lustre,lov,mdc,lquota,osc,ptlrpc,obdclass<br/> libcfs 248201 14 mds,fsfilt_ldiskfs,mgs,mgc,lustre,lov,mdc,lquota,osc,ksocklnd,ptlrpc,obdclass,lnet,lvfs<br/> ldiskfs 291319 1 fsfilt_ldiskfs<br/> mbcache 8134 2 ldiskfs,ext4<br/> jbd2 63282 3 fsfilt_ldiskfs,ldiskfs,ext4<br/> crc16 1659 2 ldiskfs,ext4<br/> root@lus04-mds1:/# lustre_rmmod <br/> open /proc/sys/lnet/dump_kernel failed: No such file or directory<br/> open(dump_kernel) failed: No such file or directory</p> <p>root@lus04-mds1:/# mount -t lustre /dev/loop2 /export/MGS<br/> root@lus04-mds1:/# mount -t lustre /dev/loop3 /export/MDT0<br/> root@lus04-mds1:/# tail /var/log/kern.log<br/> Oct 20 08:22:09 lus04-mds1 kernel: <span class="error">[66935.739538]</span> Lustre: MGS MGS started<br/> Oct 20 08:22:09 lus04-mds1 kernel: <span class="error">[66935.740321]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 20 08:22:15 lus04-mds1 kernel: <span class="error">[66942.327177]</span> LDISKFS-fs (loop3): barriers disabled<br/> Oct 20 08:22:15 lus04-mds1 kernel: <span class="error">[66942.342193]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.480158]</span> LDISKFS-fs (loop3): barriers disabled<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.481802]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.511948]</span> LustreError: 13c-e: Client log lus04-client has disappeared! Regenerating all logs.<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.514410]</span> Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.516976]</span> Lustre: Enabling user_xattr<br/> Oct 20 08:22:16 lus04-mds1 kernel: <span class="error">[66942.579581]</span> Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/loop3 with recovery enabled</p> <p>Tune the OSS and remount them</p> <p>Mount the client still fails.</p> <p>mount /lustre/scratch104<br/> mount.lustre: mount lus04-mds1@tcp0:/lus04 at /lustre/scratch104 failed: No such file or directory<br/> Is the MGS specification correct?<br/> Is the filesystem name correct?<br/> If upgrading, is the copied client log valid? (see upgrade docs)<br/> root@bc-11-2-02:~# tail /var/log/kern.log<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.917419]</span> LustreError: 23097:0:(obd_config.c:372:class_setup()) setup lus04-MDT0000-mdc-ffff88041f2c3400 failed (-2)<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.917572]</span> LustreError: 23097:0:(obd_config.c:1199:class_config_llog_handler()) Err -2 on cfg command:<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.917742]</span> Lustre: cmd=cf003 0:lus04-MDT0000-mdc 1:lus04-MDT0000_UUID 2:0@<0:0> <br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.917930]</span> LustreError: 15c-8: MGC172.17.148.4@tcp: The configuration from log 'lus04-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.918272]</span> LustreError: 23027:0:(llite_lib.c:1099:ll_fill_super()) Unable to process log: -2<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.918686]</span> LustreError: 23027:0:(obd_config.c:443:class_cleanup()) Device 2 not setup<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.918942]</span> LustreError: 23027:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.919097]</span> LustreError: 23027:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.927817]</span> Lustre: client lus04-client(ffff88041f2c3400) umount complete<br/> Oct 20 08:24:16 bc-11-2-02 kernel: <span class="error">[510327.927906]</span> LustreError: 23027:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-2)</p> <p>We have managed to mount the file system. I will update the ticket with more information once I have started the operation to copy all the data off it <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. Thanks for you help Bob</p> <p>James, Glad to hear you worked out a way to mount the file system successfully. Will be curious to see exactly what combination of settings worked for you. Looking forward to your post of more information.</p> <p>I have checked with Sven that he is happy with me posting this chat log.</p> <p>After breaking the MGS by adding the –mdt flag to it we formatted a new MGS, which is documented earlier in this bug “19/Oct/13"</p> <p>mkfs.lustre --mgs --fsname=lus04 /dev/loop2</p> <p>We tried using service nodes rather than fail node, I also suspect that mounting things on the "secondary" node helped however I can't prove this as as it was working I was loathed to break the sequence to try experiments on it.</p> <p><span class="error">[21/10/2013 10:39:55]</span> James Beal: root@lus04-mds1:/# tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.4 --servicenode=172.17.148.5 --fsname=lus04 --mgs /dev/loop2<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters:</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1174<br/> (MGS needs_index first_time update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro</p> <p><span class="error">[21/10/2013 10:51:55]</span> James Beal: root@lus04-mds1:/# tunefs.lustre --writeconf --erase-params --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --servicenode=172.17.148.4 --servicenode=172.17.148.5 --fsname=lus04 /dev/loop3<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1001<br/> (MDT no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp</p> <p> Permanent disk data:<br/> Target: lus04-MDT0000<br/> Index: 0<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1141<br/> (MDT update writeconf no_primnode )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp failover.node=172.17.148.4@tcp failover.node=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata</p> <p>Unload the lustre modules on mds1</p> <p><span class="error">[21/10/2013 10:53:56]</span> James Beal: root@lus04-mds1:/# mount -t lustre /dev/loop2 /export/MGS<br/> <span class="error">[21/10/2013 10:54:02]</span> James Beal: mount -t lustre /dev/loop3 /export/MDT0<br/> <span class="error">[21/10/2013 10:54:22]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 10:54:24]</span> James Beal: 162261.673514] LDISKFS-fs (loop3): barriers disabled<br/> <span class="error">[162261.676015]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> <span class="error">[162261.765128]</span> LDISKFS-fs (loop3): barriers disabled<br/> <span class="error">[162261.766875]</span> LDISKFS-fs (loop3): mounted filesystem with ordered data mode<br/> <span class="error">[162261.798124]</span> Lustre: MGS: Logs for fs lus04 were removed by user request. All servers must be restarted in order to regenerate the logs.<br/> <span class="error">[162261.800164]</span> Lustre: Enabling user_xattr<br/> <span class="error">[162261.801410]</span> Lustre: lus04-MDT0000: Now serving lus04-MDT0000 on /dev/loop3 with recovery enabled<br/> <span class="error">[21/10/2013 10:54:30]</span> James Beal: Shall I try and mount a client ?<br/> <span class="error">[21/10/2013 10:55:18]</span> Sven Trautmann: try a client, i'm not very confident it will work<br/> <span class="error">[21/10/2013 10:55:34]</span> James Beal: I think it is as likely to work as with the OST's<br/> <span class="error">[21/10/2013 10:55:43]</span> Sven Trautmann: :x<br/> <span class="error">[21/10/2013 10:56:35]</span> James Beal: taking its time<br/> <span class="error">[21/10/2013 10:57:22]</span> James Beal: Oct 21 10:55:49 bc-11-2-07 kernel: <span class="error">[17927384.672219]</span> Lustre: Acceptor stopping<br/> Oct 21 10:55:51 bc-11-2-07 kernel: <span class="error">[17927386.672142]</span> Lustre: Removed LNI 172.17.115.32@tcp<br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927416.866605]</span> Lustre: Build Version: v1_8_9_WC1sanger1--PRISTINE-2.6.32.59-sles-lustre-1.8.8wc1<br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927416.937734]</span> Lustre: Added LNI 172.17.115.32@tcp <span class="error">[8/256/0/180]</span><br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927416.937915]</span> Lustre: Accept secure, port 988<br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927417.028382]</span> Lustre: Lustre Client File System; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927417.048662]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 21 10:56:21 bc-11-2-07 kernel: <span class="error">[17927417.052298]</span> LustreError: 11-0: an error occurred while communicating with 172.17.148.4@tcp. The mds_connect operation failed with -11<br/> <span class="error">[21/10/2013 10:57:30]</span> James Beal: Still trying<br/> <span class="error">[21/10/2013 10:58:17]</span> Sven Trautmann: resource temporarily unavailable, sounds like missing ost<br/> <span class="error">[21/10/2013 10:58:26]</span> James Beal: Oct 21 10:58:15 bc-11-2-07 kernel: <span class="error">[17927530.816907]</span> LustreError: 11-0: an error occurred while communicating with 172.17.148.4@tcp. The mds_connect operation failed with -11<br/> <span class="error">[21/10/2013 10:58:31]</span> James Beal: How about adding one OST<br/> <span class="error">[21/10/2013 10:58:34]</span> James Beal: vd00 ?<br/> <span class="error">[21/10/2013 10:58:46]</span> James Beal: Or moving the MDS to .5 ?<br/> <span class="error">[21/10/2013 10:58:47]</span> Sven Trautmann: you did the last one already?<br/> <span class="error">[21/10/2013 10:58:54]</span> Sven Trautmann: 29<br/> <span class="error">[21/10/2013 10:59:05]</span> James Beal: Ok I will need to writeconf it again.<br/> <span class="error">[21/10/2013 10:59:10]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 10:59:25]</span> James Beal: root@lus04-mds1:/# lctl dl<br/> 0 UP mgs MGS MGS 13<br/> 1 UP mgc MGC172.17.148.4@tcp 0654a817-b2ec-a591-d828-bb850199cfe1 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> <span class="error">[21/10/2013 10:59:57]</span> James Beal: umounting all the OSS<br/> <span class="error">[21/10/2013 11:00:16]</span> Sven Trautmann: the ost where still mounted?<br/> <span class="error">[21/10/2013 11:00:22]</span> James Beal: Yes.<br/> <span class="error">[21/10/2013 11:00:32]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 11:01:11]</span> James Beal: tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/vd29<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1002<br/> (OST no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: failover.node=172.17.148.9@tcp failover.node=172.17.148.8@tcp mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp</p> <p> Permanent disk data:<br/> Target: lus04-OST001d<br/> Index: 29<br/> Lustre FS: lus04<br/> Mount type: ldiskfs<br/> Flags: 0x1142<br/> (OST update writeconf no_primnode )<br/> Persistent mount opts: errors=remount-ro,extents,mballoc<br/> Parameters: failover.node=172.17.148.9@tcp failover.node=172.17.148.8@tcp mgsnode=172.17.148.4@tcp mgsnode=172.17.148.5@tcp</p> <p>Writing CONFIGS/mountdata<br/> <span class="error">[21/10/2013 11:01:34]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 11:01:51]</span> James Beal: oot@lus04-oss4:~# mount -t lustre /dev/mapper/vd29 /export/vd29<br/> <span class="error">[21/10/2013 11:02:08]</span> James Beal: Oct 21 11:01:44 lus04-oss4 kernel: <span class="error">[247251.590225]</span> Lustre: lus04-OST001d: Now serving lus04-OST001d on /dev/mapper/vd29 with recovery enabled<br/> Oct 21 11:01:44 lus04-oss4 kernel: <span class="error">[247251.590229]</span> Lustre: lus04-OST001d: Will be in recovery for at least 5:00, or until 1 client reconnects<br/> Oct 21 11:01:48 lus04-oss4 kernel: <span class="error">[247255.743705]</span> LustreError: 20470:0:(ldlm_lib.c:887:target_handle_connect()) lus04-OST001d: NID 172.17.148.4@tcp (lus04-mdtlov_UUID) reconnected with 1 conn_cnt; cookies not random?<br/> Oct 21 11:01:48 lus04-oss4 kernel: <span class="error">[247255.743908]</span> LustreError: 20470:0:(ldlm_lib.c:887:target_handle_connect()) Skipped 7 previous similar messages<br/> Oct 21 11:01:48 lus04-oss4 kernel: <span class="error">[247255.744049]</span> LustreError: 20470:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (<del>114) req@ffff88062b1e5c00 x1449498209419298/t0 o8</del>><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1382349808 ref 1 fl Interpret:/0/0 rc -114/0<br/> Oct 21 11:01:48 lus04-oss4 kernel: <span class="error">[247255.744250]</span> LustreError: 20470:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 7 previous similar messages<br/> <span class="error">[21/10/2013 11:02:20]</span> Sven Trautmann: good<br/> <span class="error">[21/10/2013 11:02:27]</span> James Beal: root@lus04-mds1:/# lctl dl<br/> 0 UP mgs MGS MGS 9<br/> 1 UP mgc MGC172.17.148.4@tcp 0654a817-b2ec-a591-d828-bb850199cfe1 5<br/> 2 UP mdt MDS MDS_uuid 3<br/> 3 UP lov lus04-mdtlov lus04-mdtlov_UUID 4<br/> 4 UP mds lus04-MDT0000 lus04-MDT0000_UUID 3<br/> 5 UP osc lus04-OST001d-osc lus04-mdtlov_UUID 5<br/> <span class="error">[21/10/2013 11:02:32]</span> Sven Trautmann: good<br/> <span class="error">[21/10/2013 11:02:39]</span> James Beal: Client still waiting<br/> <span class="error">[21/10/2013 11:02:43]</span> James Beal: Shall I interupt it<br/> <span class="error">[21/10/2013 11:02:49]</span> James Beal: And try again ?<br/> <span class="error">[21/10/2013 11:02:54]</span> Sven Trautmann: no, the recovery needs to finish<br/> <span class="error">[21/10/2013 11:02:58]</span> James Beal: <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:03:09]</span> Sven Trautmann: what is the mds saying about recovery?<br/> <span class="error">[21/10/2013 11:03:31]</span> James Beal: Oct 21 11:02:25 lus04-mds1 kernel: <span class="error">[162781.493095]</span> Lustre: lus04-MDT0000: temporarily refusing client connection from 172.17.115.32@tcp<br/> Oct 21 11:02:25 lus04-mds1 kernel: <span class="error">[162781.493110]</span> LustreError: 12347:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (<del>11) req@ffff88061f970800 x1449498384531486/t0 o38</del>><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1382349845 ref 1 fl Interpret:/0/0 rc -11/0<br/> root@lus04-mds1:/#<br/> <span class="error">[21/10/2013 11:03:42]</span> James Beal: Which looks hopeful<br/> <span class="error">[21/10/2013 11:04:17]</span> Sven Trautmann: agree<br/> <span class="error">[21/10/2013 11:04:23]</span> James Beal: root@lus04-oss4:~# cat /proc/fs/lustre/obdfilter/lus04-OST001d/recovery_status<br/> status: RECOVERING<br/> recovery_start: 0<br/> time_remaining: 0<br/> connected_clients: 0/1<br/> delayed_clients: 0/1<br/> completed_clients: 0/1<br/> replayed_requests: 0/??<br/> queued_requests: 0<br/> next_transno: 103079215105<br/> <span class="error">[21/10/2013 11:04:39]</span> James Beal: I would be happier if that was a bit different<br/> <span class="error">[21/10/2013 11:06:15]</span> James Beal: Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171140]</span> Lustre: lus04-OST001d: Recovery period over after 0:01, of 1 clients 1 recovered and 0 were evicted.<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171145]</span> Lustre: Skipped 7 previous similar messages<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171148]</span> Lustre: lus04-OST001d: sending delayed replies to recovered clients<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171150]</span> Lustre: Skipped 7 previous similar messages<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171824]</span> Lustre: lus04-OST001d: received MDS connection from 172.17.148.4@tcp<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.171828]</span> Lustre: Skipped 6 previous similar messages<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.172491]</span> Lustre: 20474:0:(filter.c:3129:filter_destroy_precreated()) lus04-OST001d: deleting orphan objects from 50974178 to 50974199, orphan objids won't be reused any more.<br/> Oct 21 11:05:55 lus04-oss4 kernel: <span class="error">[247502.172497]</span> Lustre: 20474:0:(filter.c:3129:filter_destroy_precreated()) Skipped 6 previous similar messages<br/> <span class="error">[21/10/2013 11:06:26]</span> James Beal: cat /proc/fs/lustre/obdfilter/lus04-OST001d/recovery_status<br/> status: COMPLETE<br/> recovery_start: 1382349955<br/> recovery_duration: 0<br/> delayed_clients: 0/1<br/> completed_clients: 1/1<br/> replayed_requests: 0<br/> last_transno: 103079215104<br/> <span class="error">[21/10/2013 11:06:40]</span> James Beal: Oct 21 11:05:55 lus04-mds1 kernel: <span class="error">[162991.353819]</span> Lustre: lus04-OST001d-osc: Connection restored to service lus04-OST001d using nid 172.17.148.9@tcp.<br/> Oct 21 11:05:55 lus04-mds1 kernel: <span class="error">[162991.354453]</span> Lustre: MDS lus04-MDT0000: lus04-OST001d_UUID now active, resetting orphans<br/> <span class="error">[21/10/2013 11:06:48]</span> James Beal: ct 21 11:06:35 bc-11-2-07 kernel: <span class="error">[17928030.831136]</span> LustreError: 18685:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-513)<br/> Oct 21 11:06:35 bc-11-2-07 kernel: <span class="error">[17928030.839158]</span> Lustre: MGC172.17.148.4@tcp: Reactivating import<br/> Oct 21 11:06:35 bc-11-2-07 kernel: <span class="error">[17928030.844564]</span> Lustre: Client lus04-client(ffff880260dc8c00) mount complete<br/> <span class="error">[21/10/2013 11:07:07]</span> Sven Trautmann: nice <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:07:28]</span> James Beal: Lets get the rest of the OSS up.<br/> <span class="error">[21/10/2013 11:07:35]</span> Sven Trautmann: <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/thumbs_up.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:07:42]</span> Sven Trautmann: writeconf on all<br/> <span class="error">[21/10/2013 11:07:47]</span> James Beal: Yes<br/> <span class="error">[21/10/2013 11:07:57]</span> James Beal: A quick break <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:08:04]</span> Sven Trautmann: sure<br/> <span class="error">[21/10/2013 11:11:12]</span> James Beal: Back.<br/> <span class="error">[21/10/2013 11:12:12]</span> Sven Trautmann: ok, you do the tunefs on all OST?<br/> <span class="error">[21/10/2013 11:12:18]</span> James Beal: I will<br/> <span class="error">[21/10/2013 11:12:19]</span> James Beal: root@lus04-oss3:/# for i in `seq -w 23 28 ` ; do echo tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/$i ; done<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/23<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/24<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/25<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/26<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/27<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.9 --servicenode=172.17.148.8 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/28<br/> <span class="error">[21/10/2013 11:13:05]</span> James Beal: for i in `seq -w 15 22 ` ; do echo tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/$i ; done<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/15<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/16<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/17<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/18<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/19<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/20<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/21<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.8 --servicenode=172.17.148.9 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/22<br/> <span class="error">[21/10/2013 11:14:12]</span> Sven Trautmann: looks ok<br/> <span class="error">[21/10/2013 11:14:29]</span> James Beal: root@lus04-oss2:~# for i in `seq -w 07 14 ` ; do echo tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/$i ; done<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/07<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/08<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/09<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/10<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/11<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/12<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/13<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.6 --servicenode=172.17.148.7 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/14<br/> <span class="error">[21/10/2013 11:15:07]</span> Sven Trautmann: i guess, i'm not sure if the servicenode order has any influence at all<br/> <span class="error">[21/10/2013 11:15:13]</span> James Beal: root@lus04-oss1:~# for i in `seq -w 00 06 ` ; do echo tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/$i ; done<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/00<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/01<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/02<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/03<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/04<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/05<br/> tunefs.lustre --writeconf --erase-params --servicenode=172.17.148.7 --servicenode=172.17.148.6 --mgsnode 172.17.148.4@tcp --mgsnode 172.17.148.5@tcp --fsname=lus04 /dev/mapper/06<br/> <span class="error">[21/10/2013 11:15:21]</span> James Beal: Nor do I but I am trying to be consistent<br/> <span class="error">[21/10/2013 11:15:34]</span> James Beal: Happy with those ?<br/> <span class="error">[21/10/2013 11:17:17]</span> James Beal: ping<br/> <span class="error">[21/10/2013 11:17:37]</span> Sven Trautmann: as happy as can be<br/> <span class="error">[21/10/2013 11:17:45]</span> James Beal: ok going to do that<br/> <span class="error">[21/10/2013 11:17:55]</span> Sven Trautmann: <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/thumbs_up.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:18:16]</span> James Beal: Was missing a vd...<br/> <span class="error">[21/10/2013 11:19:19]</span> James Beal: I will start the OST mounts<br/> <span class="error">[21/10/2013 11:19:51]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 11:21:36]</span> James Beal: for i in `seq -w 01 06 `;<br/> do<br/> mount -t lustre /dev/mapper/vd$i /export/vd$i<br/> date<br/> sleep 10<br/> done<br/> Mon Oct 21 11:21:11 BST 2013<br/> Mon Oct 21 11:21:21 BST 2013<br/> Mon Oct 21 11:21:31 BST 2013<br/> <span class="error">[21/10/2013 11:22:21]</span> James Beal: root@lus04-oss2:~# for i in `seq -w 01 06 `;<br/> do<br/> mount -t lustre /dev/mapper/vd$i /export/vd$i<br/> date<br/> sleep 10<br/> done<br/> Mon Oct 21 11:21:11 BST 2013<br/> Mon Oct 21 11:21:21 BST 2013<br/> Mon Oct 21 11:21:31 BST 2013<br/> Mon Oct 21 11:21:42 BST 2013<br/> Mon Oct 21 11:21:52 BST 2013<br/> Mon Oct 21 11:22:02 BST 2013<br/> root@lus04-oss2:~#<br/> <span class="error">[21/10/2013 11:23:19]</span> Sven Trautmann: lctl dl on the client?<br/> <span class="error">[21/10/2013 11:23:41]</span> James Beal: root@bc-11-2-07:~# lctl dl<br/> 0 UP mgc MGC172.17.148.4@tcp b2a666ed-eb18-98c4-5c4e-a98d7120a06b 5<br/> 1 UP lov lus04-clilov-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 4<br/> 2 UP mdc lus04-MDT0000-mdc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 3 UP osc lus04-OST001d-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 4 UP osc lus04-OST0000-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 5 UP osc lus04-OST0001-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 6 UP osc lus04-OST0002-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 7 UP osc lus04-OST0003-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 8 UP osc lus04-OST0004-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 9 UP osc lus04-OST0005-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 10 UP osc lus04-OST0006-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 11 UP osc lus04-OST0007-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 12 UP osc lus04-OST0008-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 13 UP osc lus04-OST0009-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 14 UP osc lus04-OST000a-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> 15 UP osc lus04-OST000b-osc-ffff880260dc8c00 ddad28f4-48ec-9bb6-043d-230b0b8696f1 5<br/> <span class="error">[21/10/2013 11:23:42]</span> James Beal: <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:24:04]</span> Sven Trautmann: there are some missing?<br/> <span class="error">[21/10/2013 11:24:15]</span> James Beal: I havn't finished the mounts<br/> <span class="error">[21/10/2013 11:24:18]</span> Sven Trautmann: ok<br/> <span class="error">[21/10/2013 11:24:24]</span> James Beal: root@lus04-oss1:~# for i in `seq -w 07 14 `; do mount -t lustre /dev/mapper/vd$i /export/vd$i; date ; sleep 10; done<br/> Mon Oct 21 11:22:46 BST 2013<br/> Mon Oct 21 11:22:56 BST 2013<br/> Mon Oct 21 11:23:06 BST 2013<br/> Mon Oct 21 11:23:16 BST 2013<br/> Mon Oct 21 11:23:27 BST 2013<br/> Mon Oct 21 11:23:37 BST 2013<br/> Mon Oct 21 11:23:47 BST 2013<br/> Mon Oct 21 11:23:58 BST 2013<br/> <span class="error">[21/10/2013 11:24:37]</span> Sven Trautmann: right<br/> <span class="error">[21/10/2013 11:26:32]</span> James Beal: for i in `seq -w 15 22 `; do mount -t lustre /dev/mapper/vd$i /export/vd$i; date ; sleep 10; done<br/> Mon Oct 21 11:25:01 BST 2013<br/> Mon Oct 21 11:25:11 BST 2013<br/> Mon Oct 21 11:25:21 BST 2013<br/> Mon Oct 21 11:25:32 BST 2013<br/> Mon Oct 21 11:25:42 BST 2013<br/> Mon Oct 21 11:25:52 BST 2013<br/> Mon Oct 21 11:26:02 BST 2013<br/> Mon Oct 21 11:26:13 BST 2013<br/> root@lus04-oss3:/#<br/> <span class="error">[21/10/2013 11:26:58]</span> Sven Trautmann: recovery will take some time, i guess<br/> <span class="error">[21/10/2013 11:27:04]</span> James Beal: 5 minutes <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> <span class="error">[21/10/2013 11:27:28]</span> Sven Trautmann: if everything goes as planned, yes<br/> <span class="error">[21/10/2013 11:27:49]</span> James Beal: root@lus04-oss2:~# grep -i status /proc/fs/lustre/obdfilter/lus04-OST*/recovery_status<br/> /proc/fs/lustre/obdfilter/lus04-OST0000/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0001/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0002/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0003/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0004/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0005/recovery_status:status: COMPLETE<br/> /proc/fs/lustre/obdfilter/lus04-OST0006/recovery_status:status: COMPLETE</p> <p>I had a similar problem today on another system.<br/> For cosmetic reasons i had to change the hostnames and IPs of OSS and MDS servers.<br/> I did the following:<br/> 1. unmounted Lustre<br/> 2. changed the IP addresses of the lnet interfaces (o2ib) and<br/> 3. did a tunefs on all targets.<br/> 4. mount mgs, mdt and osts</p> <p>After the first mount i got on the MDS:</p> <p>Oct 22 16:32:34 pfs2n12 kernel: : LustreError: 28201:0:(ldlm_lib.c:383:client_obd_setup()) can't add initial connection<br/> Oct 22 16:32:34 pfs2n12 kernel: : LustreError: 28201:0:(obd_config.c:565:class_setup()) setup pfs2dat2-OST000a-osc-MDT0000 failed (-2)<br/> Oct 22 16:32:35 pfs2n12 kernel: : LustreError: 28201:0:(obd_config.c:1491:class_config_llog_handler()) Err -2 on cfg command:<br/> Oct 22 16:32:35 pfs2n12 kernel: : Lustre: cmd=cf003 0:pfs2dat2-OST000a-osc-MDT0000 1:pfs2dat2-OST000a_UUID 2:0@<0:0><br/> Oct 22 16:32:50 pfs2n12 kernel: : Lustre: 28083:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from bd8a639b-be45-6048-47b8-568404d7547d@172.26.8.19@o2ib t0 exp (null) cur 1382452370 last 0<br/> Oct 22 16:32:50 pfs2n12 kernel: : Lustre: 28083:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages</p> <p>the OSC device on the MDS was in state AT:</p> <p> 0 UP mgs MGS MGS 19<br/> 1 UP mgc MGC172.26.8.12@o2ib c4ff7eb0-8c6f-9199-45f6-f75e490ac101 5<br/> 2 UP lov pfs2dat2-MDT0000-mdtlov pfs2dat2-MDT0000-mdtlov_UUID 4<br/> 3 UP mdt pfs2dat2-MDT0000 pfs2dat2-MDT0000_UUID 3<br/> 4 UP mds mdd_obd-pfs2dat2-MDT0000 mdd_obd_uuid-pfs2dat2-MDT0000 3<br/> 5 AT osc pfs2dat2-OST000a-osc-MDT0000 pfs2dat2-MDT0000-mdtlov_UUID 1</p> <p>just like in the problem described here.</p> <p>In my case the reason for the problem was that i forgot to unload the lnet kernel module after i change the IP of the Infiniband port. The lnet id did no longer match the IP of the underlying IB interface on the OSS.<br/> After a clean lustre_rmmod the problem was gone and all OSTs could connect without problem.</p> <p>Closing old bug.</p> Development Rank 1|hzw647: Rank (Obsolete) 11116 Severity