https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-414 Lustre <p>Our admins tried to add 8 new OSS nodes to an existing lustre server cluster running 1.8.5.0-5chaos. There were 16 exiting OSS with 15 OSTs each, for a total of 240 old OSTs. There are also 15 OSTs on each of the new OSS, for a total of 120 new OSTs.</p> <p>When the new OSTs were brought up, it looks like at least 54 of the OSTs failed to be configured correctly on the MDS, and are stuck in the IN (inactive) state according to "lctl dl". I don't see a pattern to which OSTs one which new OSS failed.</p> <p>This looks similar to bug 22658 that we have seen in the past:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2011-06-14 11:57:17 LustreError: 1432:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x10612404:0x0: rc -2 2011-06-14 11:57:17 LustreError: 1432:0:(llog_obd.c:200:llog_setup()) obd lsd-OST012e-osc ctxt 2 lop_setup=ffffffff885b3dc0 failed -2 2011-06-14 11:57:17 LustreError: 1432:0:(osc_request.c:4242:osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT 2011-06-14 11:57:17 LustreError: 1432:0:(osc_request.c:4258:osc_llog_init()) osc 'lsd-OST012e-osc' tgt 'lsd-MDT0000' rc=-2 2011-06-14 11:57:17 LustreError: 1432:0:(osc_request.c:4260:osc_llog_init()) logid 0x10612404:0x0 2011-06-14 11:57:17 LustreError: 1432:0:(lov_log.c:253:lov_llog_init()) error osc_llog_init idx 302 osc 'lsd-OST012e-osc' tgt 'lsd-MDT0000' (rc=-2) 2011-06-14 11:57:17 LustreError: 1432:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x62800000028:0x10612404: rc -2 2011-06-14 11:57:17 LustreError: 1432:0:(llog_obd.c:200:llog_setup()) obd lsd-OST0130-osc ctxt 2 lop_setup=ffffffff885b3dc0 failed -2 2011-06-14 11:57:17 LustreError: 1444:0:(lov_log.c:161:lov_llog_origin_connect()) error osc_llog_connect tgt 302 (-107) 2011-06-14 11:57:17 LustreError: 1444:0:(mds_lov.c:1044:__mds_lov_synchronize()) lsd-MDT0000: lsd-OST012e_UUID failed at llog_origin_connect: -107 2011-06-14 11:57:17 Lustre: lsd-OST012e_UUID: Sync failed deactivating: rc -107 </pre> </div></div> <p>The admins decided to reboot the MDS, the MDS is still unable to activate those OSTs (at least, I assume that it is the same set of OSTs):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2011-06-14 12:49:32 LustreError: 9611:0:(lov_log.c:161:lov_llog_origin_connect()) error osc_llog_connect tgt 258 (-107) 2011-06-14 12:49:32 LustreError: 9611:0:(mds_lov.c:1044:__mds_lov_synchronize()) lsd-MDT0000: lsd-OST0102_UUID failed at llog_origin_connect: -107 2011-06-14 12:49:32 Lustre: lsd-OST0102_UUID: Sync failed deactivating: rc -107 2011-06-14 12:49:32 LustreError: 9612:0:(lov_log.c:161:lov_llog_origin_connect()) error osc_llog_connect tgt 259 (-107) 2011-06-14 12:49:32 LustreError: 9646:0:(mds_lov.c:1044:__mds_lov_synchronize()) lsd-MDT0000: lsd-OST0125_UUID failed at llog_origin_connect: -107 2011-06-14 12:49:32 LustreError: 9646:0:(mds_lov.c:1044:__mds_lov_synchronize()) Skipped 20 previous similar messages 2011-06-14 12:49:32 Lustre: lsd-OST0125_UUID: Sync failed deactivating: rc -107 </pre> </div></div> <p>Notice there is no warning about "error looking up logfile", but the lov_llog_origin_connect() is still failing.</p> <p>I suspect that lov_llog_origin_connect() is getting error code 107, ENOTCONN, from llog_obd2ops(), meaning that the llog_ctxt *ctxt is NULL. I say that, because watching the logs, I see an RPC between the MDS and OSS nodes complete successfully, but I can't see an RPC being sent after the "lov_llog_origin_connect()) connect 256/360" lines in the log.</p> <p>It appears that at the ptlrpc level, the mdt and ost are in fact fully connected. The import/export appear to be set up.</p> <p>I am beginning to suspect that the "fix" for bug 22658 that allows the mds to start up when there are missing log files just lets the server get stuck at this next point in the code.</p> <p>Also, I think there is pretty clearly some bug in Lustre's initial creation of ost llog files on the mds.</p> <p>I am attaching the mds console log for now. I can package up some more detailed lustre logs tomorrow.</p> CHAOS4.4 (RHEL5.4), lustre 1.8.5.0-5chaos LU-414

error looking up logfile

Bug Major Resolved Fixed Hongchao Zhang Christopher Morrone llnl Tue, 14 Jun 2011 20:47:17 +0000 Wed, 11 Oct 2017 19:55:38 +0000 Wed, 11 Oct 2017 19:55:38 +0000 Lustre 2.4.0 0 8 <p>Hongchao</p> <p>Could you please look at this one?</p> <p>Thanks</p> <p>Peter</p> <p>this error is a little different from the 22658, <br/> In this issue, the CATLOG of some OSC doesn't exist, but there is a record in the CATLOG of the MDT, then it cause "llog_setup" to fail in "osc_llog_init".<br/> in 22658, the CATLOG of the OSC does exist, but the actual plain log files(containing the real operation log info) doesn't exist.</p> <p>and there is still two "error looking up logfile" after the MDS reboot in your attached console log,<br/> ...<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x10612404:0x0: rc -2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(llog_obd.c:200:llog_setup()) obd lsd-OST012e-osc ctxt 2 lop_setup=ffffffff885aedc0 failed -2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4242:osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4258:osc_llog_init()) osc 'lsd-OST012e-osc' tgt 'lsd-MDT0000' rc=-2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4260:osc_llog_init()) logid 0x10612404:0x0<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(lov_log.c:253:lov_llog_init()) error osc_llog_init idx 302 osc 'lsd-OST012e-osc' tgt 'lsd-MDT0000' (rc=-2)<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(llog_lvfs.c:612:llog_lvfs_create()) error looking up logfile 0x62800000028:0x10612404: rc -2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(llog_obd.c:200:llog_setup()) obd lsd-OST0130-osc ctxt 2 lop_setup=ffffffff885aedc0 failed -2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4242:osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4258:osc_llog_init()) osc 'lsd-OST0130-osc' tgt 'lsd-MDT0000' rc=-2<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(osc_request.c:4260:osc_llog_init()) logid 0x62800000028:0x10612404<br/> 2011-06-14 12:46:52 LustreError: 8762:0:(lov_log.c:253:lov_llog_init()) error osc_llog_init idx 304 osc 'lsd-OST0130-osc' tgt 'lsd-MDT0000' (rc=-2)<br/> ...</p> <p>Are the same count of OST stuck in the IN (inactive) state after you reboot the MDS?</p> <p>for the llog, I'm afraid whether there is a problem which some uninitialized data is read in the MDS's CATLOG? <br/> Are the index of these newly added OSTs larger than the existing ones?</p> <p>Yes, as far as I can tell they are all larger than existing ones. OSTs 0000-00ef are on the original 16 nodes, and 00f0-0167 are on the new 8 nodes.</p> <p>I do believe that it was the same inactive OSTs after reboot. But the logs squash some messages, making it a little hard to verify 100%. The sysadmin is pretty certain that it was the same count of OSTs that was inactive before and after reboot, and some spot checking of the logs verifies at least those OST messages that are not squashed are the same.</p> <p>See the new attachment. I translated the MDS's CATALOGS file into a readable form, and it is pretty clear which entries are incorrect. Any that look like "10612404:0" or "64000000028:1580334" are bad. They have no matching file in the OBJECTS directory, and they correspond with the OSTs that are stuck in the inactive state.</p> <p>yes, the MDS's CATALOGS is corrupted! and I suspect it is caused by the disorderly call of mds_lov_update_desc for the<br/> newly added OSTs, if one OST with larger index is initialized before those with small index, there will leave an <br/> uninitialized area in the log file, which cause this issue.</p> <p>Hi Chris,<br/> Could you please try to delete these insane records in the MDS's CATALOGS to check whether the problem is fixed? Thanks!</p> <p>btw, I have written the following patch and will push it to Gerrit after testing it locally,</p> <p>diff --git a/lustre/obdclass/llog_lvfs.c b/lustre/obdclass/llog_lvfs.c<br/> index 43d88a3..e6d6534 100644<br/> &#8212; a/lustre/obdclass/llog_lvfs.c<br/> +++ b/lustre/obdclass/llog_lvfs.c<br/> @@ -833,9 +833,10 @@ int llog_put_cat_list(struct obd_device *obd, struct obd_device *disk_obd,</p> { struct lvfs_run_ctxt saved; struct l_file *file; + void *buf = NULL; int rc, rc1 = 0; int size = sizeof(*idarray) * count; - loff_t off = idx * sizeof(*idarray); + loff_t filesize, off = idx * sizeof(*idarray); if (!count) GOTO(out1, rc = 0); @@ -856,6 +857,17 @@ int llog_put_cat_list(struct obd_device *obd, struct obd_device *disk_obd, GOTO(out, rc = -ENOENT); } <p>+ filesize = i_size_read(file-&gt;f_dentry-&gt;d_inode);<br/> + if (filesize &lt; off) </p> { + loff_t count = off - filesize; + OBD_ALLOC(buf, count); + if (buf == NULL) + GOTO(out, rc = -ENOMEM); + + fsfilt_write_record(disk_obd, file, buf, count, &amp;filesize, 1); + OBD_FREE(buf, count); + } <p>+<br/> rc = fsfilt_write_record(disk_obd, file, idarray, size, &amp;off, 1);<br/> if (rc) {<br/> CDEBUG(D_INODE,"OBD filter: error writeing %s: rc %d\n",</p> <p>Yes, I'll zero the records and see if it will work. We have a downtime scheduled for Tuesday, June 21, so I will try it then.</p> <p>the rhel5 kernel version for b1_8 has been updated recently(update to 2.6.18-238.12.1), and some time is spent on<br/> preparing the new build&amp;test environment, then the local test for the patch is delayed, and it could be complete tomorrow.</p> <p>the patch is at <a href="http://review.whamcloud.com/#change,987" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,987</a></p> <p>Zeroing the CATALOGS file got everything working this morning.</p> <p>as per the code line, the newly allocated block in the CATLOG file is initialized with zero in the current b1_8 and master,<br/> and is also the case for v1_8_5, then it should not a use-uninitialized-data issue, but the content of the CATLOG does <br/> indicate it is, and the first insane data appears just beyond 8192 (the 256th entry). </p> <p>Hi Chris,<br/> Is the block size in MDT's device 8192? and could you please help to paste your source code tree here? Thanks!</p> <p><a href="https://github.com/chaos/lustre/tree/1.8.5.0-5chaos" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/tree/1.8.5.0-5chaos</a></p> <p>I will need to check the block size.</p> <p>in osc_llog_init, "llog_get_cat_list" and "llog_put_cat_list" is protected by the mutex "obd_llog_cat_process" of MDT,<br/> then the problem should not be related to the race between OSCs, the remaining possible case is the problem during "fsfilt"<br/> writing the record into log file.</p> <p>Hi Chris,<br/> Is your rhel5.4 the official one? I want to do some tests on lustre 1.8.5.0-5chaos.</p> <p>No, it is CHAOS4.4, so some changes from RHEL5.4.</p> <p>The MDS is using a 3ware card and the array is configured as RAID10. Stripe size is 64k.</p> <p>Hi Chris,</p> <p>could you please tell me where can I get the CHAOS4.4? Thanks!</p> <p>I have tested the 1.8.5.0-5chaos against 2.6.18-194.17.1, it used 2 OSS(one has 252 OSTs, the other has 50 OSTs) and<br/> no such problem was triggered.</p> <p>FYI, I have encountered an error "EXTRA_DIST: variable 'bin_SCRIPTS' is used but 'bin_SCRIPTS' is undefined" during <br/> running "sh autogen.sh", and I skipped it and no other issues are shown up in following compile&amp;test.</p> <p><a href="ftp://gdo-lc.ucllnl.org/pub/projects/chaos/" class="external-link" target="_blank" rel="nofollow noopener">ftp://gdo-lc.ucllnl.org/pub/projects/chaos/</a></p> <p>Can you tell me more about your test? Why did you have the OST count so imbalanced?</p> <p>We saw this with only 15 OSTs on each of 8 OSS, so I wouldn't think such a high count would be necessary. Did you statically allocate the index numbers to the osts, and alternate them between nodes? Did you issue the mount commands in parallel?</p> <p>the test is run on my two virtual nodes using loop device, one node runs MDT+(252 OSTs), the other runs 50 OSTs<br/> the I don't specify the index value of these OSTs and don't alternate them, the mount commands aren't in parallel.</p> <p>the reason of testing it in this case is the operations of CATALOGS "llog_get_cat_list" and "llog_put_cat_list", <br/> both are protected by the mutex "obd_llog_cat_process" of MDT, then the problem is the "fsfilt" layer during writing<br/> the record into log file.</p> <p>Ah, I think thats part of your problem.</p> <p>We DO specify our index values. If you aren't doing that, then the index numbers will just be sequentially allocated as the OSTs connect the first time. And if you are not mounting the first time in parallel, then there is no chance to reproduce the issue we are seeing.</p> <p>I specify the index of the OSTs and mount them not sequentially, the problem still not occurs. and I can't mount these OSTs<br/> in parallel in my test environment, and could you please help to test it?</p> <p>Chris &#8211; I'm doing some cleanup work on JIRA issues. <br/> Would you prefer that I keep this open and unresolved? <br/> Or may I mark it as 'resolved &#8211; cannot reproduce'?<br/> Many thanks,<br/> ~ jfc.</p> <p>I think it is very likely that this is still a problem. We seem to hit it in every version of Lustre. Our admins now have a permanent rule of manually adding OSTs sequentially to Lustre because this code is so broken. That is not acceptable. It turns an operation that should take under a minute into one that takes well over an hour.</p> <p>As of b2_4 (starting to use LOD/OSP instead of LOV/OSC at MDT), this issue was fixed in "osp_sync_llog_init", it will recreate<br/> the log file if it didn't exist previously (previously it only create the log file if the log ID is zero)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> osp_sync_llog_init(<span class="code-keyword">const</span> struct lu_env *env, struct osp_device *d) { ... ctxt = llog_get_context(obd, LLOG_MDS_OST_ORIG_CTXT); LASSERT(ctxt); <span class="code-keyword">if</span> (likely(logid_id(&amp;osi-&gt;osi_cid.lci_logid) != 0)) { rc = llog_open(env, ctxt, &amp;lgh, &amp;osi-&gt;osi_cid.lci_logid, NULL, LLOG_OPEN_EXISTS); <span class="code-comment">/* re-create llog <span class="code-keyword">if</span> it is missing */</span> <span class="code-keyword">if</span> (rc == -ENOENT) logid_set_id(&amp;osi-&gt;osi_cid.lci_logid, 0); <span class="code-keyword">else</span> <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out_cleanup, rc); } <span class="code-keyword">if</span> (unlikely(logid_id(&amp;osi-&gt;osi_cid.lci_logid) == 0)) { rc = llog_open_create(env, ctxt, &amp;lgh, NULL, NULL); <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out_cleanup, rc); osi-&gt;osi_cid.lci_logid = lgh-&gt;lgh_id; } LASSERT(lgh != NULL); ctxt-&gt;loc_handle = lgh; rc = llog_cat_init_and_process(env, lgh); <span class="code-keyword">if</span> (rc) GOTO(out_close, rc); rc = llog_osd_put_cat_list(env, d-&gt;opd_storage, d-&gt;opd_index, 1, &amp;osi-&gt;osi_cid); ... } </pre> </div></div> Development End date Fri, 18 Jul 2014 20:47:17 +0000 Rank 1|hzw0vb: Rank (Obsolete) 10221 Severity Start date Tue, 14 Jun 2011 20:47:17 +0000