Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-2110] Unable to mount (-17) MDT https://jira.whamcloud.com/browse/LU-2110 Lustre <p>After updating to 2.3.53-2chaos, the MDS is no longer able to mount its MDT. The relevant console messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Found index 0 for lstest-MDT0000, updating log LustreError: 33410:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context LustreError: 33836:0:(genops.c:316:class_newdev()) Device lstest-MDT0000-osp-MDT0000 already exists at 136, won't add LustreError: 33836:0:(obd_config.c:374:class_attach()) Cannot create device lstest-MDT0000-osp-MDT0000 of type osp : -17 Lustre: lstest-MDT0000: Temporarily refusing client connection from 0@lo LustreError: 11-0: lstest-MDT0000-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11 LustreError: 33836:0:(obd_mount.c:373:lustre_start_simple()) lstest-MDT0000-osp-MDT0000 attach error -17 LustreError: 33836:0:(obd_mount.c:1135:lustre_osp_setup()) lstest-MDT0000-osp-MDT0000: setup up failed: rc -17 LustreError: 15c-8: MGC172.20.5.2@o2ib500: The configuration from log 'lstest-client' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 33405:0:(obd_mount.c:1865:server_start_targets()) lstest-MDT0000: failed to start OSP: -17 Lustre: lstest-MDT0000: Unable to start target: -17 Lustre: Failing over lstest-MDT0000 LustreError: 32689:0:(client.c:1116:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880faa7f0800 x1415277549978464/t0(0) o13-&gt;lstest-OST0181-osc-MDT0000@172.20.2.185@o2ib500:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 LustreError: 32690:0:(osp_precreate.c:116:osp_statfs_interpret()) lstest-OST0182-osc-MDT0000: couldn't update statfs: rc = -5 LustreError: 32689:0:(client.c:1116:ptlrpc_import_delay_req()) Skipped 253 previous similar messages Lustre: server umount lstest-MDT0000 complete LustreError: 33405:0:(obd_mount.c:2985:lustre_fill_super()) Unable to mount (-17) </pre> </div></div> <p>I'm just about to start looking into the root cause.</p> Lustre: 2.3.53-2chaos LU-2110 Unable to mount (-17) MDT Bug Critical Resolved Fixed Alex Zhuravlev Prakash Surya Mon, 8 Oct 2012 12:43:59 +0000 Fri, 19 Apr 2013 16:25:04 +0000 Fri, 19 Apr 2013 16:25:04 +0000 Lustre 2.4.0 Lustre 2.4.0 0 7 <p>is this on the clean node (after reboot) ? can you unload all lustre modules (may take some time) and try again ?</p> <p>Originally, this this was on a clean reboot. But the messages I pasted in the description were from a manually retried mount, after the first failed.</p> <p>Here are all the messages from the console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Lustre: Build Version: 2.3.53-2chaos-2chaos--PRISTINE-2.6.32-220.23.1.1chaos.ch5.x86_64 Lustre: Found index 0 for lstest-MDT0000, updating log LustreError: 32758:0:(mgc_request.c:248:do_config_log_add()) failed processing sptlrpc log: -2 LustreError: 32761:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context LustreError: 33225:0:(genops.c:316:class_newdev()) Device lstest-MDT0000-osp-MDT0000 already exists at 136, won't add LustreError: 33225:0:(obd_config.c:374:class_attach()) Cannot create device lstest-MDT0000-osp-MDT0000 of type osp : -17 Lustre: lstest-MDT0000: Temporarily refusing client connection from 0@lo LustreError: 11-0: lstest-MDT0000-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11 LustreError: 33225:0:(obd_mount.c:373:lustre_start_simple()) lstest-MDT0000-osp-MDT0000 attach error -17 LustreError: 33225:0:(obd_mount.c:1135:lustre_osp_setup()) lstest-MDT0000-osp-MDT0000: setup up failed: rc -17 LustreError: 15c-8: MGC172.20.5.2@o2ib500: The configuration from log 'lstest-client' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 32758:0:(obd_mount.c:1865:server_start_targets()) lstest-MDT0000: failed to start OSP: -17 Lustre: lstest-MDT0000: Unable to start target: -17 Lustre: Failing over lstest-MDT0000 LustreError: 32680:0:(client.c:1116:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880f977bb800 x1415277549977961/t0(0) o13-&gt;lstest-OST0181-osc-MDT0000@172.20.2.185@o2ib500:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 LustreError: 32680:0:(osp_precreate.c:116:osp_statfs_interpret()) lstest-OST0181-osc-MDT0000: couldn't update statfs: rc = -5 LustreError: 32681:0:(client.c:1116:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff882016d06c00 x1415277549977962/t0(0) o13-&gt;lstest-OST0182-osc-MDT0000@172.20.2.186@o2ib500:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 LustreError: 32682:0:(osp_precreate.c:116:osp_statfs_interpret()) lstest-OST0183-osc-MDT0000: couldn't update statfs: rc = -5 LustreError: 32682:0:(osp_precreate.c:116:osp_statfs_interpret()) Skipped 1 previous similar message LustreError: 32680:0:(osp_precreate.c:116:osp_statfs_interpret()) Skipped 125 previous similar messages Lustre: server umount lstest-MDT0000 complete LustreError: 32758:0:(obd_mount.c:2985:lustre_fill_super()) Unable to mount (-17) Lustre: Found index 0 for lstest-MDT0000, updating log LustreError: 33410:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context LustreError: 33836:0:(genops.c:316:class_newdev()) Device lstest-MDT0000-osp-MDT0000 already exists at 136, won't add LustreError: 33836:0:(obd_config.c:374:class_attach()) Cannot create device lstest-MDT0000-osp-MDT0000 of type osp : -17 Lustre: lstest-MDT0000: Temporarily refusing client connection from 0@lo LustreError: 11-0: lstest-MDT0000-osp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11 LustreError: 33836:0:(obd_mount.c:373:lustre_start_simple()) lstest-MDT0000-osp-MDT0000 attach error -17 LustreError: 33836:0:(obd_mount.c:1135:lustre_osp_setup()) lstest-MDT0000-osp-MDT0000: setup up failed: rc -17 LustreError: 15c-8: MGC172.20.5.2@o2ib500: The configuration from log 'lstest-client' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 33405:0:(obd_mount.c:1865:server_start_targets()) lstest-MDT0000: failed to start OSP: -17 Lustre: lstest-MDT0000: Unable to start target: -17 Lustre: Failing over lstest-MDT0000 LustreError: 32689:0:(client.c:1116:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880faa7f0800 x1415277549978464/t0(0) o13-&gt;lstest-OST0181-osc-MDT0000@172.20.2.185@o2ib500:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 LustreError: 32690:0:(osp_precreate.c:116:osp_statfs_interpret()) lstest-OST0182-osc-MDT0000: couldn't update statfs: rc = -5 LustreError: 32689:0:(client.c:1116:ptlrpc_import_delay_req()) Skipped 253 previous similar messages Lustre: server umount lstest-MDT0000 complete LustreError: 33405:0:(obd_mount.c:2985:lustre_fill_super()) Unable to mount (-17) </pre> </div></div> <p>Is is still worth rebooting, and trying again?</p> <p>well, sorry you're seeing this... could you please try again and attach lustre log to the ticket ?</p> <p>Rebooted and collected the lustre log file.</p> <p>thanks. I see the root cause.. working on the fix.</p> <p>Prakash, please try with <a href="http://review.whamcloud.com/#change,4227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4227</a> </p> <p>if I understand right, failover nid for MDS was specified at mkfs.lustre time, not added later ?</p> <p>Actually, I'm not certain of that. At one point a failover NID was added using a writeconf, but the filesystem was reformatted since then. During the reformat, I'm unsure if both the failover NIDs were specified at mkfs time, or the writeconf method was used after mkfs. I can try to track down that information if it is useful..?</p> <p>one way is to fetch /CONFIGS/lstest-client file from MDS and parse it with llog_reader utility.<br/> it would help us if you attach it to the ticket as well. thanks.</p> <p>Here you go. The dump of</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># grove-mds2 /mnt/grove-mds2/mgs &gt; llog_reader CONFIGS/lstest-client &gt; lstest-client.llogreader </pre> </div></div> <p>#12 (128)attach 0:lstest-MDT0000-mdc 1:mdc 2:lstest-clilmv_UUID<br/> ... <br/> #20 (088)add_uuid nid=172.20.2.185@o2ib500(0x501f4ac1402b9) 0: 1:172.20.2.185@o2ib500 <br/> #21 (088)add_uuid nid=172.20.2.185@tcp(0x20000ac1402b9) 0: 1:172.20.2.185@o2ib500 </p> <p>#21 resulted in a second instance of OSP device.</p> <p>I think the patch above should help with the issue.</p> <p>Ok, we've pulled in patch <a href="http://review.whamcloud.com/#change,4227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4227</a> and will give it a try.</p> <p>Patch landed to master.</p> <p>can we close the ticket?</p> <p>Sure.</p> Development Rank 1|hzv6u7: Rank (Obsolete) 4642 Severity