https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-730 Lustre <p><span class="error">&#91;root@dc-mds01 log&#93;</span># mount -t lustre /dev/vg_dc/mdt /lustre/dc/mdt<br/> mount.lustre: mount /dev/vg_dc/mdt at /lustre/dc/mdt failed: Invalid argument<br/> This may have multiple causes.<br/> Are the mount options correct?<br/> Check the syslog for more info.</p> <p><span class="error">&#91;root@dc-mds01 log&#93;</span># tail /var/log/messages<br/> Oct 3 16:00:07 dc-mds01 multipathd: dm-7: add map (uevent)<br/> Oct 3 16:00:21 dc-mds01 crm_shadow: <span class="error">&#91;11747&#93;</span>: info: Invoked: crm_shadow <br/> Oct 3 16:00:21 dc-mds01 crm_resource: <span class="error">&#91;11748&#93;</span>: info: Invoked: crm_resource --meta -r mgs -p target-role -v Started <br/> Oct 3 16:00:22 dc-mds01 multipathd: dm-6: umount map (uevent)<br/> Oct 3 16:00:32 dc-mds01 crm_shadow: <span class="error">&#91;11970&#93;</span>: info: Invoked: crm_shadow <br/> Oct 3 16:00:32 dc-mds01 crm_resource: <span class="error">&#91;11971&#93;</span>: info: Invoked: crm_resource --meta -r mdt_dc -p target-role -v Started <br/> Oct 3 16:23:10 dc-mds01 crm_shadow: <span class="error">&#91;12965&#93;</span>: info: Invoked: crm_shadow <br/> Oct 3 16:23:10 dc-mds01 crm_resource: <span class="error">&#91;12966&#93;</span>: info: Invoked: crm_resource --meta -r mdt_dc -p target-role -v Started <br/> Oct 3 16:39:36 dc-mds01 multipathd: dm-7: umount map (uevent)<br/> Oct 3 16:39:39 dc-mds01 multipathd: dm-7: umount map (uevent)</p> <p><span class="error">&#91;root@dc-mds01 log&#93;</span># tail /var/log/kern<br/> May 13 04:41:13 dc-mds01 kernel: <span class="error">&#91;132780.597275&#93;</span> Lustre: 12660:0:(client.c:1482:ptlrpc_expire_one_request()) Skipped 6 previous similar messages<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.375612&#93;</span> Lustre: 12660:0:(quota_master.c:1716:mds_quota_recovery()) Only 28/30 OSTs are active, abort quota recovery<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.375619&#93;</span> Lustre: 12660:0:(quota_master.c:1716:mds_quota_recovery()) Skipped 2 previous similar messages<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.375626&#93;</span> Lustre: dc-OST001b-osc: Connection restored to service dc-OST001b using nid 149.165.235.234@tcp9.<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.375629&#93;</span> Lustre: Skipped 2 previous similar messages<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.376540&#93;</span> Lustre: MDS dc-MDT0000: dc-OST001b_UUID now active, resetting orphans<br/> May 13 04:41:16 dc-mds01 kernel: <span class="error">&#91;132783.376542&#93;</span> Lustre: Skipped 2 previous similar messages<br/> May 13 04:41:40 dc-mds01 kernel: <span class="error">&#91;132808.132985&#93;</span> Lustre: 12660:0:(quota_master.c:1716:mds_quota_recovery()) Only 28/30 OSTs are active, abort quota recovery<br/> May 13 04:41:40 dc-mds01 kernel: <span class="error">&#91;132808.133028&#93;</span> Lustre: dc-OST001d-osc: Connection restored to service dc-OST001d using nid 149.165.235.234@tcp9.<br/> May 13 04:41:40 dc-mds01 kernel: <span class="error">&#91;132808.133921&#93;</span> Lustre: MDS dc-MDT0000: dc-OST001d_UUID now active, resetting orphans</p> LU-730

MDS Won't Mount

Bug Minor Resolved Fixed Oleg Drokin Stephen Simms Mon, 3 Oct 2011 16:43:01 +0000 Mon, 3 Oct 2011 17:45:11 +0000 Mon, 3 Oct 2011 17:45:11 +0000 0 1 <p>dmesg output:<br/> <span class="error">&#91;13466.908788&#93;</span> LDISKFS-fs (dm-7): warning: maximal mount count reached, running e2fsck is recommended<br/> <span class="error">&#91;13466.913958&#93;</span> LDISKFS-fs (dm-7): mounted filesystem with ordered data mode<br/> <span class="error">&#91;13466.965528&#93;</span> LDISKFS-fs (dm-7): warning: maximal mount count reached, running e2fsck is recommended<br/> <span class="error">&#91;13466.970602&#93;</span> LDISKFS-fs (dm-7): mounted filesystem with ordered data mode<br/> <span class="error">&#91;13466.985515&#93;</span> Lustre: Enabling user_xattr<br/> <span class="error">&#91;13466.991612&#93;</span> Lustre: 13691:0:(mds_fs.c:677:mds_init_server_data()) RECOVERY: service dc-MDT0000, 4 recoverable clients, 0 delayed clients, last_transno 85899346187<br/> <span class="error">&#91;13466.995427&#93;</span> Lustre: dc-MDT0000: Now serving dc-MDT0000 on /dev/vg_dc/mdt with recovery enabled<br/> <span class="error">&#91;13466.995430&#93;</span> Lustre: dc-MDT0000: Will be in recovery for at least 5:00, or until 4 clients reconnect<br/> <span class="error">&#91;13466.995928&#93;</span> Lustre: 13691:0:(lproc_quota.c:448:lprocfs_quota_wr_type()) dc-MDT0000: quotaon failed because quota files don't exist, please run quotacheck firstly<br/> <span class="error">&#91;13466.995936&#93;</span> Lustre: dc-MDT0000.mdt: set parameter quota_type=ug<br/> <span class="error">&#91;13466.996197&#93;</span> Lustre: 13691:0:(mds_lov.c:1155:mds_notify()) MDS dc-MDT0000: add target dc-OST0000_UUID<br/> <span class="error">&#91;13466.996201&#93;</span> Lustre: 13691:0:(mds_lov.c:1155:mds_notify()) Skipped 28 previous similar messages<br/> <span class="error">&#91;13467.031748&#93;</span> LustreError: 13691:0:(obd_config.c:979:class_process_proc_param()) Can't parse param nosquash_nids<br/> <span class="error">&#91;13467.031753&#93;</span> LustreError: 13691:0:(obd_config.c:1199:class_config_llog_handler()) Err -22 on cfg command:<br/> <span class="error">&#91;13467.031758&#93;</span> Lustre: cmd=cf00f 0:dc-MDT0000 1:mdt.nosquash_nids <br/> <span class="error">&#91;13467.031903&#93;</span> LustreError: 15b-f: MGC149.165.235.235@tcp: The configuration from log 'dc-MDT0000' failed (-22). Make sure this client and the MGS are running compatible versions of Lustre.<br/> <span class="error">&#91;13467.031908&#93;</span> LustreError: 15c-8: MGC149.165.235.235@tcp: The configuration from log 'dc-MDT0000' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> <span class="error">&#91;13467.031914&#93;</span> LustreError: 13571:0:(obd_mount.c:1126:server_start_targets()) failed to start server dc-MDT0000: -22<br/> <span class="error">&#91;13467.031921&#93;</span> LustreError: 13571:0:(obd_mount.c:1655:server_fill_super()) Unable to start targets: -22<br/> <span class="error">&#91;13467.031939&#93;</span> Lustre: Failing over dc-MDT0000<br/> <span class="error">&#91;13467.031941&#93;</span> Lustre: Skipped 30 previous similar messages<br/> <span class="error">&#91;13467.048812&#93;</span> Lustre: dc-MDT0000: shutting down for failover; client state will be preserved.<br/> <span class="error">&#91;13467.049987&#93;</span> Lustre: MDT dc-MDT0000 has stopped.<br/> <span class="error">&#91;13468.686492&#93;</span> LustreError: 137-5: UUID 'dc-MDT0000_UUID' is not available for connect (no target)<br/> <span class="error">&#91;13468.686501&#93;</span> LustreError: 13622:0:(ldlm_lib.c:1914:target_send_reply_msg()) @@@ processing error (<del>19) req@ffff811809545800 x1379230428949522/t0 o38</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 368/0 e 0 to 0 dl 1317674478 ref 1 fl Interpret:/0/0 rc -19/0<br/> <span class="error">&#91;13470.046025&#93;</span> Lustre: server umount dc-MDT0000 complete<br/> <span class="error">&#91;13470.046033&#93;</span> LustreError: 13571:0:(obd_mount.c:2050:lustre_fill_super()) Unable to mount (-22)</p> <p>Oleg will look into this one</p> <p>I think this is the case of bugzilla bug 14693. - invalid config param preventing a service from starting<br/> IT's a pretty old bug and I don't know what sort fo version you are running.<br/> There was another reopening in 2010 with some more patches in bug 17471.</p> <p>The new functionality should now allow you to delete the invalid parameters with option -d too.</p> <p>btw the offending parameter is mdt.nosquash_nids </p> <p>Ok, to further comment on this, the parameter is actually valid parameter, but it's a lustre 2.1 thing.<br/> Somebody was playing with 2.1 stuff and typed something in a wrong terminal window?</p> <p>Ok, to further track how and when the parameter was set, I just checked the code, there is this printing in mgs_wlp_lcfg:<br/> LCONSOLE_INFO("%sing parameter %s.%s in log %s\n", del ? "Disabl" : rc ?<br/> "Sett" : "Modify", tgtname, comment, logname);</p> <p>This is output into the kernel logs, so if you have functional kernel logs on the MGS node, search for this line that contains squash in it.</p> <p>(I noticed that the kernlog file you pasted was quite stale, hopefully you have another source of kernel messages that is valid).</p> <p>Ashley Pittman at DDN just got back to us with the following:</p> <p>lctl conf_param -d dc-MDT0000.mdt.nosquash_nids</p> <p>and we can now mount metadata. You guys nailed the culprit and now we're in the know.</p> <p>Thanks as always for your help!<br/> Simms</p> <p>Glad to hear it Steve - marking the ticket as resolved</p> Development Rank 1|hzvhvz: Rank (Obsolete) 6547 Severity