|
This is caused by OBD_INCOMPAT_MULTI_RPCS being set on the MDS. It should be cleared if the MDS is unmounted cleanly.
|
|
before downgrade the system, the script did cleanupall to umount
|
|
Hit the same issue when downgrading from master RHEL7 to 2.5.5 RHEL6.6.
before downgrade, the MDS is unmounted
|
|
Gregoire
Could you please advise on this one?
Thanks
Peter
|
|
To have the OBD_INCOMPAT_MULTI_RPCS incompatiblity flag cleaned, the MDT target must have no client connected when it is unmounted.
Connected clients include both Lustre clients and other MDT targets.
Therefore, if the file system has only one MDT target, unmounting the Lustre clients first will allow clearing the incompatibility flag at MDT target unmount.
If the file system has several MDT targets, then it is required to unmount all MDT targets, and then, for each MDT target (one by one), mount with abort_recovery option and unmount the target. This is mentioned in the LU-5319 test plan, at section "Upgrade / Downgrade".
After clearing the incompatiblity flag, the server can be downgraded to a lower Lustre version.
|
|
Hi Gregoire,
I hit this issue when doing a clean downgrade from master to 2.5.5, which umount all the servers and clients and downgrade them all at once, then tried to mount the system again and failed.
I will try with 2.7.0 and see how it goes.
|
|
Sarah,
The issue will be the same with a downgrade to 2.7.0, if you don't perform the additional operation that clears the incompatibility flag.
The important point is that to have the OBD_INCOMPAT_MULTI_RPCS incompatibility flag cleared on the MDT servers, it must unmount all the clients, then unmount the servers, and then additionally operate the "mount with abort_recovery option and unmount of each MDT one by one". After that the nodes can be downgraded to a lower Lustre version.
|
|
Ah I see thank you for the clarification!
|
|
If I understand correctly, this is not a bug
|
|
Hello Gregoire,
I hit the same issue recently, on master/tag-2.7.66 and b2_8/tag-2.7.90. I did remount the MDS with option "abort_recovery" and umount it again before downgrading, here is what I saw. The same test passed on tag-2.7.64, do you have any idea why this happens?
on MDS
dmesg of MDS
|
|
Could you provide the complete test case that was executed ?
How is designed the filesystem (nodes hosting the MGT, MDTs, OSTs, client nodes...) ?
It would be helpful to also provide the full MDS lustre log, including the log before downgrade and log after downgrade.
thanks.
|
|
the complete case is:
1. format and setup the system with 1 MDS(1MDT), 1 OSS(1 OST) and 2 clients with lustre 2.5.5 RHEL6.6; create some data
2. shundown the whole system, umount all nodes
3. upgrade the whole system to b2_8/build #8; only clear the boot disk, keep data disk untouched
4. remount the whole system, check the data, works fine;
5. shudown the whole system again, umount all nodes
6. do additional step, remounting the MDS with abort_recovery option
7. umount the MDS again
8. downgrade all servers and clients to 2.5.5 again without touching the data disk
9. mount MDS failed as above.
Please find the attached for more logs. 'before means before downgrade; after means after downgrade'
|
|
MDS logs before and after downgrade
update: I tried today with b2_8/build #11, manually ran those steps without using script, it doesn't hit the problem.
|
|
Could you add to the script some calls to the command "lr_reader <mdt-target-device>" at different places (between steps 5-6, 6-7, 7-8 and 8-9 for example) ?
This could help identify why the incompatibility flag is not cleared.
The output of the command looks like this:
The flag OBD_INCOMPAT_MULTI_RPCS = 0x400 can be checked within feature_incompat value.
|
|
Hi Gregoire,
I performed a basic clean upgrading/downing testing from EE 2.4.2.2 (tag 2.5.42.15) to EE 3.0.1.0 (tag 2.7.16.5) with the following steps:
- format and mount EE 2.4.2.2 filesystem with 1 MGS/MDS (1 MDT), 1 OSS (1 OST) and 1 Client
- unmount the whole filesystem
- upgrade the whole filesystem to EE 3.0.1.0
- mount the whole filesystem
- unmount the whole filesystem
- remount MDS and OSS with "-o abort_recovery" option
- unmount MDS and OSS
- downgrade the whole filesystem to EE 2.4.2.2
- mount MDS still failed:
|
|
Hi, Sarah
How did you umount the MDT in 7th step?
6. do additional step, remounting the MDS with abort_recovery option
7. umount the MDS again
If you didn't use "umount -f", could you try with "umount -f" to see if the problem can be reproduced?
|
|
Hi Niu,
No I didn't use "-f", I will try today and get back to you. thank you!
|
|
Thank you YuJian for the information.
Niu, I tried with "-f" option(step 7) and it seems work, upgrade from EE2.4.2.2 RHEL6.8 to EE3.0.1 RHEL7.2 and downgrade again:
MDS
I also record the lc_reader value for all steps for reference:
1. first time mount system under EE2.4.2.2 RHEL6.8
2. umount MDS
3. after upgrade to EE3.0.1 RHEL7 and remount
4. umount again
5. remount with abort_recovery
6. umount with "-f"
7. downgrade the system to EE2.4.2.2 and mount again
|
|
I think that we can safely close this out from a community release point of view. Upgrade/downgrade from 2.5.x to 2.9 is outside the official scope of the release and there is a viable workaround for those who want to try this anyway.
|
Generated at Sat Feb 10 02:08:40 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
debug-after