https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3302 Lustre <p>We updated a client to 2.3.64-4chaos and tried to mount a 2.3.63-6chaos server. The mount fails with</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 15c-8: MGC172.20.20.201@o2ib500: The configuration from log 'fsv-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 14351:0:(llite_lib.c:1043:ll_fill_super()) Unable to process log: -2 Lustre: Unmounted fsv-client LustreError: 14351:0:(obd_mount.c:1265:lustre_fill_super()) Unable to mount (-2) </pre> </div></div> <p>Using git bisect I found the mount failure was introduced with this patch:</p> <p><a href="http://review.whamcloud.com/#change,5820" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5820</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-2684 fid: unify ostid and FID </pre> </div></div> <p>The critical questions at this point are:</p> <ul class="alternate" type="square"> <li>Can we solve this problem by updating both server and client to 2.3.64-4chaos?</li> </ul> <ul class="alternate" type="square"> <li>Can we safely upgrade the server, or does the above patch introduce on-disk format incompatibilities?</li> </ul> <ul class="alternate" type="square"> <li>Will we be able to safely revert the server to 2.3.63 in case we find problems, or will it write new objects in an incompatible format?</li> </ul> <p>LLNL-bug-id: TOSS-2060</p> PPC client LU-3302

ll_fill_super() Unable to process log: -2

Bug Blocker Resolved Fixed Di Wang Ned Bass LB Thu, 9 May 2013 00:29:59 +0000 Fri, 17 May 2013 16:58:04 +0000 Fri, 10 May 2013 18:17:23 +0000 Lustre 2.4.0 Lustre 2.4.0 0 6 <p>As an editorial comment, while we understand that interoperability issues are inevitable in a pre-release branch, we wish such changes would be advertised more prominently. Clear statements about compatibility between tags would really help us plan our update process. At a minimum, patches that introduce incompatibilities should say so clearly in the commit message.</p> <p>Di, can you advise us on this? Thanks</p> <p>Ned, can you please attach a -1 debug log from the 2.3.64 client, and ideally also from the MGS.</p> <p>I agree that the <a href="https://jira.whamcloud.com/browse/LU-2684" title="convert ost_id to lu_fid for FID_SEQ_NORMAL objects" class="issue-link" data-issue-key="LU-2684"><del>LU-2684</del></a> change was problematic, and it was intended to only change the network protocol between clients and OSTs when running DNE. The <a href="https://jira.whamcloud.com/browse/LU-2888" title="After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed" class="issue-link" data-issue-key="LU-2888"><del>LU-2888</del></a> patch <a href="http://review.whamcloud.com/6044" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6044</a> (which was already included in 2.3.64) should have fixed the LLOG handling, so I'm not sure what the exact cause of your problem is. AFAIK, the current master code interoperates with 2.1.5 and 2.3.0 properly, but there might be something specific with your setup that is causing grief.</p> <p>Andreas, yes I'll grab the logs.</p> <p>Note the above error was a 2.3.64 client talking to a 2.3.63 server. Do you mean that patch 6044 fixed LLOG handling on the client, or is it needed on the server as well?</p> <p>Attaching -1 debug logs for client and MDS. Note these were not captured from the same mount attempt.</p> <p>The NID of the client is 172.20.16.10@o2ib500.</p> <p>I did notice the mgs got ENOENT handling opcodes LLOG_ORIGIN_HANDLE_CREATE and LLOG_ORIGIN_HANDLE_READ_HEADER:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>20000000:01000000:6.0:1368040430.786604:0:18265:0:(mgs_handler.c:757:mgs_handle()) @@@ MGS fail to handle opc = 501: rc = -2 req@ffff881019ecb050 x1434494006460492/t0(0) o501-&gt;2e89e428-68d9-71a1-75f0-147bc1963566@172.20.16.10@o2ib500:0/0 lens 296/0 e 0 to 0 dl 1368040491 ref 1 fl Interpret:/0/ffffffff rc 0/-1 ... 20000000:01000000:6.0:1368040430.788063:0:18265:0:(mgs_handler.c:757:mgs_handle()) @@@ MGS fail to handle opc = 503: rc = -2 req@ffff881019f14850 x1434494006460504/t0(0) o503-&gt;2e89e428-68d9-71a1-75f0-147bc1963566@172.20.16.10@o2ib500:0/0 lens 272/0 e 0 to 0 dl 1368040491 ref 1 fl Interpret:/0/ffffffff rc 0/-1 </pre> </div></div> <p>The 6044/<a href="https://jira.whamcloud.com/browse/LU-2888" title="After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed" class="issue-link" data-issue-key="LU-2888"><del>LU-2888</del></a> patch fixed the handling on the server, but the original problem patch from <a href="https://jira.whamcloud.com/browse/LU-2684" title="convert ost_id to lu_fid for FID_SEQ_NORMAL objects" class="issue-link" data-issue-key="LU-2684"><del>LU-2684</del></a> wasn't in 2.3.63, so it shouldn't be relevant.</p> <p>Di</p> <p>Could you please comment on this?</p> <p>Thanks</p> <p>Peter</p> <p>Ned, is this a PPC client? It would be useful to include this information in the "Environment" section when filing a bug.</p> <p>Yes it is. Sorry for the omission.</p> <p>John, it was mentioned to me that you have already found some endian issues with the FID-on-OST code? Could you please point out where they are, it might be that this is the source of the problem being seen here, since we didn't see any problems with our x86_64 clients for interoperability.</p> <p>Possibly. Please see <a href="https://jira.whamcloud.com/browse/LU-3294" title="osp_sync_llog_init(): ASSERTION( lgh != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-3294"><del>LU-3294</del></a>.</p> <p>Ned, it would be interesting to know what happens when you create a new 2.3.65 FS on ppc, unmount, and then remount it.</p> <p>John, okay, we're getting a test environment set up where I should be able to do that test.</p> <p>Haven't tried 2.3.65 yet, but initial testing suggests updating the server to 2.3.64 lets the mount succeed. Here's what I did:</p> <p>1. Tried to mount a 2.3.62 server from a 2.3.64 PPC client. Fails with "ll_fill_super() Unable to process log: -2"<br/> 2. Updated the server to 2.3.64. Mount from 2.3.64 PPC client succeeds.</p> <p>Ned, I just checked the debug log, it seems client get correct log ID after swab, </p> <p>Here are the client log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000040:00000001:5.0:1368040600.989913:5152:8187:0:(llog_swab.c:86:lustre_swab_llogd_body()) Process entered 00000040:00001000:5.0:1368040600.989914:5328:8187:0:(llog_swab.c:53:print_llogd_body()) llogd body: c000000f50e9a100 00000040:00001000:5.0:1368040600.989915:5328:8187:0:(llog_swab.c:55:print_llogd_body()) lgd_logid.lgl_oi: 0x6400000000000000:16777216 00000040:00001000:5.0:1368040600.989915:5328:8187:0:(llog_swab.c:56:print_llogd_body()) lgd_logid.lgl_ogen: 0x0 00000040:00001000:5.0:1368040600.989916:5328:8187:0:(llog_swab.c:57:print_llogd_body()) lgd_ctxt_idx: 0x0 00000040:00001000:5.0:1368040600.989917:5328:8187:0:(llog_swab.c:58:print_llogd_body()) lgd_llh_flags: 0x0 00000040:00001000:5.0:1368040600.989917:5328:8187:0:(llog_swab.c:59:print_llogd_body()) lgd_index: 0x0 00000040:00001000:5.0:1368040600.989918:5328:8187:0:(llog_swab.c:60:print_llogd_body()) lgd_saved_index: 0x0 00000040:00001000:5.0:1368040600.989918:5328:8187:0:(llog_swab.c:61:print_llogd_body()) lgd_len: 0x0 00000040:00001000:5.0:1368040600.989919:5328:8187:0:(llog_swab.c:62:print_llogd_body()) lgd_cur_offset: 0x0 00000040:00001000:5.0:1368040600.989920:5328:8187:0:(llog_swab.c:53:print_llogd_body()) llogd body: c000000f50e9a100 00000040:00001000:5.0:1368040600.989920:5328:8187:0:(llog_swab.c:55:print_llogd_body()) lgd_logid.lgl_oi: 0x64:1 00000040:00001000:5.0:1368040600.989921:5328:8187:0:(llog_swab.c:56:print_llogd_body()) lgd_logid.lgl_ogen: 0x0 00000040:00001000:5.0:1368040600.989921:5328:8187:0:(llog_swab.c:57:print_llogd_body()) lgd_ctxt_idx: 0x0 00000040:00001000:5.0:1368040600.989922:5328:8187:0:(llog_swab.c:58:print_llogd_body()) lgd_llh_flags: 0x0 00000040:00001000:5.0:1368040600.989923:5328:8187:0:(llog_swab.c:59:print_llogd_body()) lgd_index: 0x0 00000040:00001000:5.0:1368040600.989923:5328:8187:0:(llog_swab.c:60:print_llogd_body()) lgd_saved_index: 0x0 00000040:00001000:5.0:1368040600.989924:5328:8187:0:(llog_swab.c:61:print_llogd_body()) lgd_len: 0x0 00000040:00001000:5.0:1368040600.989924:5328:8187:0:(llog_swab.c:62:print_llogd_body()) lgd_cur_offset: 0x0 00000040:00000001:5.0:1368040600.989925:5152:8187:0:(llog_swab.c:97:lustre_swab_llogd_body()) Process leaving </pre> </div></div> <p>But somehow server can not find the log object by this ID. Unfortunately, I can not find correspondent mgs handling information in the MDS debug log. Could you please redo the test update the debug log. </p> <p>In the mean time, I do see there are some problem during the logid swab(John also point out one in <a href="https://jira.whamcloud.com/browse/LU-3294" title="osp_sync_llog_init(): ASSERTION( lgh != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-3294"><del>LU-3294</del></a>) I will cook the patch now. </p> <p><a href="http://review.whamcloud.com/#change,6305" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6305</a></p> <p>Ned would you confirm that these are x86_64 servers and ppc/ppc64 clients? In that case it's unlikely that you're affected by <a href="https://jira.whamcloud.com/browse/LU-3294" title="osp_sync_llog_init(): ASSERTION( lgh != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-3294"><del>LU-3294</del></a> since that issue is probably limited to BE servers.</p> <p>Yes these are x86_64 servers and ppc64 clients.</p> <p>Also, if it is an unfixed swabbing bug, I would expect the mount to also fail with 2.3.64 servers.</p> <p>Now that this patch has landed, can we get confirmation that this is fixed?<br/> Thank you!</p> <p>With the patch, a 2.3.64 PPC client can mount from a 2.3.63 server. So this appears to be fixed. Thanks</p> <p>Based on latest comments, this patch landed and has fixed the issue. Closing ticket.</p> Related LU-3294 Development Rank 1|hzvqhr: Rank (Obsolete) 8173 Severity