https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4384 Lustre <p>Before upgrade, server and client are running 2.5.0 ldiskfs, then upgrade the whole system to lustre-master build #1791, it passed; then downgrade the system to 2.5.0 again, when mounting the OST, got following error:</p> <p>OST console shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == upgrade-downgrade Lustre version and system information == 11:31:49 (1386963109) Lustre: Lustre: Build Version: 2.5.0-RC1--PRISTINE-2.6.32-358.18.1.el6_lustre.x86_64 LNet: Added LNI 10.10.19.53@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: DEBUG MARKER: == upgrade-downgrade End == 11:31:52 (1386963112) LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. quota=on. Opts: LustreError: 33847:0:(ofd_fs.c:588:ofd_server_data_init()) lustre-OST0000: unsupported incompat filesystem feature(s) 10 LustreError: 33847:0:(obd_config.c:572:class_setup()) setup lustre-OST0000 failed (-22) LustreError: 33847:0:(obd_config.c:1591:class_config_llog_handler()) MGC10.10.19.62@tcp: cfg command failed: rc = -22 Lustre: cmd=cf003 0:lustre-OST0000 1:dev 2:0 3:f LustreError: 15b-f: MGC10.10.19.62@tcp: The configuration from log 'lustre-OST0000'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. LustreError: 15c-8: MGC10.10.19.62@tcp: The configuration from log 'lustre-OST0000' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 33730:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server lustre-OST0000: -22 LustreError: 33730:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -22 LustreError: 33730:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) lustre-MDT0000-lwp-OST0000: Can't end config log lustre-client. LustreError: 33730:0:(obd_mount_server.c:1426:server_put_super()) lustre-OST0000: failed to disconnect lwp. (rc=-2) LustreError: 33730:0:(obd_config.c:619:class_cleanup()) Device 3 not setup Lustre: server umount lustre-OST0000 complete LustreError: 33730:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount /dev/sdb1 (-22) [root@wtm-88 ~]# </pre> </div></div> <br/> LU-4384

Hit unsupported incompat filesystem feature error after downgrade system from 2.6 to 2.5.0

Bug Blocker Resolved Fixed Mikhail Pershin Sarah Liu MB mn4 Fri, 13 Dec 2013 19:52:55 +0000 Thu, 22 May 2014 18:37:11 +0000 Mon, 17 Mar 2014 19:11:53 +0000 Lustre 2.4.0 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.6.0 Lustre 2.5.2 0 6 <p>The EXT4_FEATURE_INCOMPAT_METABG = 0x10. I don't know why this would be set when upgrading to 2.6. This feature relates to filesystem resizing but should not be enabled by default. What kernel is used for 2.6? It looks like 2.6.32-358.18.1 is used for 2.5. </p> <p>the kernel used in 2.6 is 2.6.32-358.23.2</p> <p><a href="http://build.whamcloud.com/job/lustre-master/1791/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1791/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/</a></p> <p>Sorry, I misunderstood the problem above. This is not an EXT4_INCOMPAT_META_BG flag, this is actually OBD_INCOMPAT_FID being set in the last_rcvd header by 2.6.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/** FID is enabled */</span> #define OBD_INCOMPAT_FID 0x00000010 </pre> </div></div> <p>Strangely, I don't see OBD_INCOMPAT_FID being set in OFD_INCOMPAT_SUPP on master, nor OFD_INCOMPAT_SUPP being used anywhere. It seems this checking has moved over to tgt_lastrcvd.c::tgt_scd[] as part of the unified target patches in <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> (<a href="http://review.whamcloud.com/7330" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7330</a>). I can't yet see how OBD_INCOMPAT_FID is being set, but this is a definite problem for downgrade.</p> <p>Actually, it looks like tgt_server_data_init() is unconditionally setting OBD_INCOMPAT_FID on all filesystems. This should be limited to MDT filesystems. What is the meaning of OBD_INCOMPAT_FID on an OST filesystem anyway? Is that for FID_SEQ_NORMAL objects being allocated there?</p> <p>rolling downgrade also hit this problem on OST</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: Added LNI 10.10.19.53@tcp [8/256/0/180] LNet: Accept secure, port 988 LNet: 9095:0:(debug.c:218:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. LNet: 9096:0:(debug.c:218:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 48 MB is too small for debug buffer size, setting it to 128 MB. LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. quota=on. Opts: LustreError: 9242:0:(ofd_fs.c:588:ofd_server_data_init()) lustre-OST0000: unsupported incompat filesystem feature(s) 10 LustreError: 9242:0:(obd_config.c:572:class_setup()) setup lustre-OST0000 failed (-22) LustreError: 9242:0:(obd_config.c:1591:class_config_llog_handler()) MGC10.10.19.62@tcp: cfg command failed: rc = -22 Lustre: cmd=cf003 0:lustre-OST0000 1:dev 2:0 3:f LustreError: 15b-f: MGC10.10.19.62@tcp: The configuration from log 'lustre-OST0000'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. LustreError: 15c-8: MGC10.10.19.62@tcp: The configuration from log 'lustre-OST0000' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 9125:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server lustre-OST0000: -22 LustreError: 9125:0:(obd_mount_server.c:1732:server_fill_super()) Unable to start targets: -22 LustreError: 9125:0:(obd_mount_server.c:848:lustre_disconnect_lwp()) lustre-MDT0000-lwp-OST0000: Can't end config log lustre-client. LustreError: 9125:0:(obd_mount_server.c:1426:server_put_super()) lustre-OST0000: failed to disconnect lwp. (rc=-2) LustreError: 9125:0:(obd_config.c:619:class_cleanup()) Device 3 not setup Lustre: server umount lustre-OST0000 complete LustreError: 9125:0:(obd_mount.c:1311:lustre_fill_super()) Unable to mount /dev/sdb1 (-22) [root@wtm-88 ~]# </pre> </div></div> <p><a href="http://review.whamcloud.com/8810" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8810</a></p> <p>patch to remove unconditional set of OBD_INCOMPAT_FID for all types of filesystems. Now it is set for MDT only as before.</p> <p>Can this ticket be closed now that Change, 8810 has landed or is more work needed in this ticket?</p> <p>I think some more work is still needed to make this handling correct:</p> <ul class="alternate" type="square"> <li>add a patch for b2_4, b2_5, and master to add OBD_INCOMPAT_FID to OFD_INCOMAP_SUPP</li> <li>fix checking of OFD_INCOMAT_SUPP in b2_5 and master</li> <li>set OBD_INCOMPAT_FID on OSTs when FID_SEQ_NORMAL objects are created</li> </ul> <p>Ok, are we safe to reduce this from blocker at this point? Or do we need to continue to track as a 2.6 blocker until these remaining tasks are completed?</p> <p>I think all three issues are still blockers for 2.6, and #1 back porting the fix to 2.4.3 and 2.5.1 is a blocker there, so that users can downgrade again. </p> <p>Andreas, if server supports FID_SEQ_NORMAL and create such objects, does that mean the old client will be incompatible? I had impression that client will work anyway with FID being just converted to OID/SEQ format. So I wonder do we have here incompatible case at all?</p> <p>Mike, the use of OBD_INCOMPAT_FID is completely separate from any client interoperability. That is needed to prevent old OST code from mounting the filesystem after it has started to create FID_SEQ_NORMAL objects that the 2.3 and older server does not understand.</p> <p>patches to set OBD_INCOMPAT_FID bit:</p> <p>master - <a href="http://review.whamcloud.com/9375" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9375</a><br/> b2_4 - <a href="http://review.whamcloud.com/9410" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9410</a><br/> b2_5 - <a href="http://review.whamcloud.com/9411" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9411</a></p> <p>The checking fix is needed only in master and it is landed.</p> <p>Landed for master. Fixes for maintenance branches will be tracked separately.</p> Related LU-3467 Development Rank 1|hzwbdj: Rank (Obsolete) 12019 Severity