https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8044 Lustre <p>On startup for the first time after formatting, the MDT fails to process the config provided by the MGS. The MDT then fails to start.<br/> The config log on the MGS appears to be invalid, with more than one setup and modify_mdc_tgt record for one of the other MDTs.</p> <p>The MDT which fails to start reports:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Lustre: Build Version: 2.8.0 LustreError: 11797:0:(obd_config.c:1262:class_process_config()) no device for: lustre-MDT0021-mdtlov LustreError: 11797:0:(obd_config.c:1666:class_config_llog_handler()) MGC192.168.112.240@o2ib15: cfg command failed: rc = -22 Lustre: cmd=cf014 0:lustre-MDT0021-mdtlov 1:lustre-MDT0014_UUID 2:20 3:1 LustreError: 15b-f: MGC192.168.112.240@o2ib15: The configuration from log 'lustre-MDT0021'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. LustreError: 11667:0:(obd_mount_server.c:1309:server_start_targets()) failed to start server lustre-MDT0021: -22 LustreError: 11667:0:(obd_mount_server.c:1798:server_fill_super()) Unable to start targets: -22 LustreError: 11667:0:(obd_mount_server.c:1512:server_put_super()) no obd lustre-MDT0021 Lustre: server umount lustre-MDT0021 complete LustreError: 11667:0:(obd_mount.c:1426:lustre_fill_super()) Unable to mount (-22) </pre> </div></div> <p>The config logs CONFIGS/lustre-MDT* do not all have the same number of records. lustre-MDT0021 has 2 more records than the other 29 MDTs.</p> <p>The suspicious llog records are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#04 (152)setup 0:lustre-MDT0014-osp-MDT0021 1:lustre-MDT0014_UUID 2:192.168.113.6@o2ib15 #05 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov 1:lustre-MDT0014_UUID 2:20 3:1 #179 (152)setup 0:lustre-MDT0014-osp-MDT0021 1:lustre-MDT0014_UUID 2:192.168.113.6@o2ib15 #180 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov 1:lustre-MDT0014_UUID 2:20 3:1 </pre> </div></div> TOSS 2 (RHEL 6.7 based)<br/> kernel 2.6.32-573.22.1.1chaos.ch5.4.x86_64<br/> Lustre 2.8.0+patches 2.8-llnl-preview1<br/> zfs-0.6.5.4-1.ch5.4.x86_64<br/> 1 MGS - separate server<br/> 40 MDTs - each on separate server<br/> 10 OSTs - each on separate server<br/> Filesystem name is &quot;lustre&quot; LU-8044

class_process_config() no device for: lustre-MDT0021-mdtlov

Bug Major Resolved Fixed Di Wang Olaf Faaland llnl Tue, 19 Apr 2016 17:59:34 +0000 Thu, 14 Jun 2018 21:41:18 +0000 Wed, 15 Jun 2016 13:15:25 +0000 Lustre 2.8.0 Lustre 2.9.0 0 7 <p>My description made it sound like this happens every time. That's not the case; it happens intermittently.</p> <p>Attached:<br/> config log for MDT0021 from CONFIGS/ on MGS<br/> console log (dmesg) for MDS (catalyst240)<br/> ldev.conf showing what nodes play what roles<br/> lctl dk output from MGS, reflects default debug and subsystem_debug settings</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#04 (152)setup 0:lustre-MDT0014-osp-MDT0021 1:lustre-MDT0014_UUID 2:192.168.113.6@o2ib15 #05 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov 1:lustre-MDT0014_UUID 2:20 3:1 </pre> </div></div> <p>The index for OSP setup seems too earlier, which does not look right. </p> <p>Could you please post CONFIGS/lustre-MDT0021 and CONFIG/lustre-MDT0000 here? thanks.</p> <p>Config log for MDT0000.<br/> The log for MDT0021 is already attached.</p> <p>Di,</p> <p>For the next few hours, I can either gather more information for you, or experiment. About 4 hours from now, I'll have to put the nodes back to their production use and the filesystem will be destroyed.</p> <p>thanks,<br/> Olaf</p> <p>According to the config log, it looks OSP (lustre-MDT0014-osp-MDT0021) setup record is added before "lov setup", which is clearly wrong. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#01 (224)marker 865 (flags=0x01, v2.8.0.0) lustre-MDT0014 'add osp' Tue Apr 19 08:55:48 2016- #02 (088)add_uuid nid=192.168.113.6@o2ib15(0x5000fc0a87106) 0: 1:192.168.113.6@o2ib15 #03 (144)attach 0:lustre-MDT0014-osp-MDT0021 1:osp 2:lustre-MDT0021-mdtlov_UUID #04 (152)setup 0:lustre-MDT0014-osp-MDT0021 1:lustre-MDT0014_UUID 2:192.168.113.6@o2ib15 #05 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov 1:lustre-MDT0014_UUID 2:20 3:1 #06 (224)END marker 865 (flags=0x02, v2.8.0.0) lustre-MDT0014 'add osp' Tue Apr 19 08:55:48 2016- #07 (224)marker 873 (flags=0x01, v2.8.0.0) lustre-MDT0021 'add mdt' Tue Apr 19 08:55:48 2016- #08 (120)attach 0:lustre-MDT0021 1:mdt 2:lustre-MDT0021_UUID #09 (112)mount_option 0: 1:lustre-MDT0021 2:lustre-MDT0021-mdtlov #10 (160)setup 0:lustre-MDT0021 1:lustre-MDT0021_UUID 2:33 3:lustre-MDT0021-mdtlov 4:f #11 (224)END marker 873 (flags=0x02, v2.8.0.0) lustre-MDT0021 'add mdt' Tue Apr 19 08:55:48 2016- </pre> </div></div> <p>I am checking the debug log on MGS to see why this happen. </p> <p>Olaf, Could you please try to reproduce the log with debug level = -1 on MGS? it will help me to figure out what happens there. thanks.</p> <p>Ah, it looks like a race when MGS register 2 MDTs at the same time, I will cook a patch.</p> <p>Attach debug log from MGS with debug = -1, while MDTs coming up for first time.</p> <p>In this log, MDT0002 (on catalyst243, NID 192.168.112.243@o2ib15) encountered the error.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Lustre: Build Version: 2.8.0 LustreError: 11826:0:(obd_config.c:1262:class_process_config()) no device for: lustre-MDT0002-mdtlov LustreError: 11826:0:(obd_config.c:1666:class_config_llog_handler()) MGC192.168.112.240@o2ib15: cfg command failed: rc = -22 Lustre: cmd=cf014 0:lustre-MDT0002-mdtlov 1:lustre-MDT0023_UUID 2:35 3:1 LustreError: 15b-f: MGC192.168.112.240@o2ib15: The configuration from log 'lustre-MDT0002'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. LustreError: 11696:0:(obd_mount_server.c:1309:server_start_targets()) failed to start server lustre-MDT0002: -22 LustreError: 11696:0:(obd_mount_server.c:1798:server_fill_super()) Unable to start targets: -22 LustreError: 11696:0:(obd_mount_server.c:1512:server_put_super()) no obd lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 11696:0:(obd_mount.c:1426:lustre_fill_super()) Unable to mount (-22) </pre> </div></div> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19658" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19658</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8044" title="class_process_config() no device for: lustre-MDT0021-mdtlov" class="issue-link" data-issue-key="LU-8044"><del>LU-8044</del></a> mgs: Only add OSP for registered MDT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3ccd18da205192ec0ad527ec88b69793aa5e6670</p> <p>Olaf: the new debug log seems not catch the failure, probably too late or -1 make the dk log too big to catch all of information? But anyway the patch 19658 should help here. Please try this when you have another chance. Thanks.</p> <p>Hi Di,<br/> Assigning to you as I see you have already commented and provided a fix in a new patch.<br/> Thanks.<br/> Joe</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19658/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19658/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8044" title="class_process_config() no device for: lustre-MDT0021-mdtlov" class="issue-link" data-issue-key="LU-8044"><del>LU-8044</del></a> mgs: Only add OSP for registered MDT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c67a74b55c126ec1be6c195cb2e8cb8c2e6cf868</p> <p>patch has landed to master for 2.9.0</p> Related Development End date Fri, 3 Jun 2016 17:59:34 +0000 Rank 1|hzy8mn: Rank (Obsolete) 9223372036854775807 Severity Start date Tue, 19 Apr 2016 17:59:34 +0000