Logs are on FTP at ftp.whamcloud.com:/uploads/LU-6050/LU-6050_20141218.tar.gz
The MDS-side problem looks like it would be fixed by http://review.whamcloud.com/12617 "LU-5778 lod: Fix lod_qos_statfs_update()" since it is failing when the OST is down.
There is a separate question of why the OST is down. This appears to be caused by:
osd_check_lma() perses1-OST0005: FID [0x100000000:0x0:0x0] != self_fid [0x100050000:0x0:0x0]
This is because master uses the "correct" IDIF FID for OST index 5 (SEQ=0x100000000 | (ost_idx << 16)) while the old code used the "wrong" fid (only storing OST index 0 in the IDIF FID).
The object in question looks to be the O/0/LAST_ID object (OID=0, VER=0 is LAST_ID for a particular IDIF sequence number). You may be able to fix this problem by mounting the OST as type ldiskfs, and deleting the "trusted.lma" xattr (either using setfattr -d "trusted.lma" O/0/LAST_ID, or copying it to a temporary file and then renaming it back on top of the original (which will work as long as "cp" doesn't copy xattrs). Did you run in DNE mode on master, or run LFSCK on the OST0005?
A quick look at the commit log shows this might relate to http://review.whamcloud.com/11304 part (2), or http://review.whamcloud.com/10580, or http://review.whamcloud.com/9560, or http://review.whamcloud.com/7053 so we need to see what can be done to avoid this, and whether there is already a patch in b2_5 that will avoid this problem?
Andreas,
Thanks for the response.
Good to understand what that object is. (It was clearly something special, but I couldn't quite tell which special object it was or find it on the underlying volume.)
I've just tried the suggested workaround and can confirm that worked to get the file system up and running. Thank you very much. 
If you don't mind, could you elaborate on what's stored in that xattr and how the file system recovered from having it deleted?
We did NOT run in DNE mode on master (just one MDT on this file system), and while I didn't deliberately run LFSCK on OST0005, I note that the OI scrub was run automatically. I have log messages of it trying to run and failing. (Note that we have this same error on all of the OSTs. There were similar messages for OST0005, these are just the errors I have handy.)
2014-12-18T10:53:07.060468-06:00 c1-0c1s4n2 Lustre: perses1-OST0003-: trigger OI scrub by RPC for [0x100000000:0x491c8a1:0x0], rc = 0 [1]
2014-12-18T10:53:07.060502-06:00 c1-0c1s4n2 LustreError: 2975:0:(ldlm_resource.c:1185:ldlm_resource_get()) perses1-OST0003: lvbo_init failed for resource 0x491c8a1:0x0: rc = -115
2014-12-18T10:53:07.615746-06:00 c0-0c0s2n1 Lustre: perses1-OST0004-: trigger OI scrub by RPC for [0x100000000:0x4ab46ca:0x0], rc = 0 [1]
2014-12-18T10:53:07.615771-06:00 c0-0c0s2n1 Lustre: Skipped 9 previous similar messages
2014-12-18T10:53:07.615778-06:00 c0-0c0s2n1 LustreError: 2943:0:(ldlm_resource.c:1185:ldlm_resource_get()) perses1-OST0004: lvbo_init failed for resource 0x4ab46ca:0x0: rc = -115
The trusted.lma (struct lustre_mdt_attrs) xattr stores the "self" FID for each object. In older OST filesystems this was stored in the trusted.fid xattr along with the MDT inode's FID, but in 2.4 the back-end code between OST and MDT were unified. The old OST code didn't have any concept of what OST index it was using, so it always stored "f_seq=0" into the LMA FID. One of the patches referenced in my previous comment changed the OST code to store this FID as a proper IDIF FID, which includes the OST index into the FID so that it is unique across all OSTs, instead of always re-using OST index 0. That resolved some issues with LFSCK checking consistency between the MDT and OST objects.
We've done upgrade/downgrade testing previously, so it isn't clear why this problem wasn't seen in our local testing, or on the other OSTs in your test system. There were some changes made to the code to allow the old and new style of IDIF FIDs to compare properly, which likely needs to be backported to b2_5, but I think Wang Di and Fan Yong recall the details better than I do.
on the other OSTs in your test system
Actually, this hit all the OSTs on our test system. Sorry, I may not have made that clear.
One thing I forgot to mention - This is a system that was upgraded from 1.8 to 2.4, then from 2.4 to 2.5. The file system was originally formatted with 1.8. (Dirdata was added and quotas updated.)
Thanks for the detailed explanation. I'm a bit concerned by the idea that users need to patch 2.5 in order to be able to downgrade - That sort of defeats the idea of backwards compatibility, doesn't it?
The issue you hit is that when your system was upgraded to master, the master generated some new-formatted IDIF (which contains the real OST index) and stored such IDIF in the OST-object's LMA EA. There are three cases that may cause the new-formtted IDIF generated:
1) New created the OST-object after the upgrading. Currently, master will generate new-formatted IDIF for the new created OST-obejct.
2) For the old OST-object that was created before the upgrading, the OI scrub will convert the old-formatted IDIF to new-formatted IDIF.
3) For the old OST-object that was created before the upgrading, even though OI scrub is not run, when the OST-object is accessed, the system will auto-check whether its up-layer given FID (new formatted IDIF) matches the self-FID in LMA EA (old formatted IDIF) or not, if not match, then conditionally converted the self-FID as new-formatted IDIF. There is a proc switch under osd-ldiskfs can be used to control that: lma_self_repair, which is enabled by default.
If the OST-object contains new-formatted IDIF in its LMV EA, in spite of it is new created or converted one, then after the system downgraded to Lustre-2.5, the old system will found that the up layer given FID (old formatted IDIF) does not match the self-FID in LMV EA (new formatted IDIF). Because there is no further compatible handling on b2_5, then it will case kinds of unexpected failures.
To resolve the compatibility issues, there are two possible solutions:
1) We can make some patch for b2_5 to recognize the new-formatted IDIF.
2) From the long view, maybe we need to control the new-formatted IDIF generating as handling "dirdata", once the administrator specified "tune2fs -O new-idif", then the OSD-ldiskfs will generate new-formatted IDIF or convert old-formatted IDIF, and NOT allow to downgrade any longer.
nasf, since we've effectively created an incompatibility preventing downgrades, I think your second option is the way to go. (Unless there is some other way to resolve this.)
I think in general it's not OK to make downgrade impossible without the user taking specific action. Your second suggestion would provide that.
The tune2fs tool is not the right mechanism for this. The tune2fs utility is for ext2/3/4 filesystems only, and has nothing to do with the problem here. This needs to be done with the feature flags in the last_rcvd file, that can be set for newly formatted filesystems, or manually set via tunefs.lustre or maybe when a manual LFSCK is run on the OST?
I also recommend that we make a patch for b2_5 if this problem was present in 2.6.0, since we are not making any updates for 2.6.x. It is bad for the code to be fragile in the face of a minor inconsistency on disk and should be able to handle this case, and you can bet that someone will want to downgrade even after they "knew" they didn't have to anymore. I doubt that 2.5.3 is the end of the road for any 2.5 installation, so we may as well fix this problem in both places.
Note: the new ROCOMPAT flag in the OSTs should be in the SUPPORTED mask for 2.7, and can be added to the SUPPORTED mask for 2.5 when it is patched, so that downgrades are safe. The b2_5 patch should NOT set the ROCOMPAT flag in the last_rcvd file, since this would cause problems for 2.5.x interop.
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/13516
Subject: LU-6050 target: control OST-index in IDIF via ROCOMPAT flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 03ba2e91a4cd4363dcb5cf4561155c111581ad21
Making this one a blocker so it is landed for 2.7. Otherwise it isn't possible to downgrade to 2.5 after an upgrade.
Note that it would also be possible to backport support for the OST index in the IDIF FIDs to 2.5 in order to allow downgrades after an upgrade, but that isn't a requirement for this to land to 2.7 and is only mentioned here for future reference.
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/13516/
Subject: LU-6050 target: control OST-index in IDIF via ROCOMPAT flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ecd28d9b6cb691bda8184a7e07f1acc1ccded391
Landed for 2.7