[LU-9960] Rolling downgrade master to 2.10 - unable to import zpools

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0, Lustre 2.10.1, Lustre 2.11.0
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Type: Bug
• Priority: Critical
• Reporter: Saurabh Tandan (Inactive)
• Assignee: Nathaniel Clark
• Resolution: Fixed

Environment

master, 3631 zfs 0.7.1
2.10, build 5 zfs 0.6.5.9
ZFS

• Severity: 3

Description

While performing Rolling downgrade testing for bfs from master to 2.10, I am unable to import the zpool.

[root@onyx-78 ~]# zpool import lustre-mdt1
[  939.565075] SPL: using hostid 0x61303230
This pool uses the following feature(s) not supported by this system:
	org.zfsonlinux:userobj_accounting (User/Group object accounting.)
All unsupported features are only required for writing to the pool.
The pool can be imported using '-o readonly=on'.
cannot import 'lustre-mdt1': unsupported version or feature

Steps Followed:

1. Started with all clients and servers with 2.10 build 5 and build the Lustre File system
2. Upgraded OSS to master re-mounted and ran sanity.sh
3. Upgraded MDS to master , remounted and ran sanity.sh
4. Upgraded Clients to master, remounted and ran sanity.sh
5. Downgraded Clients to 2.10. remounted and ran sanity.sh
6. Downgraded MDS to 2.10, while importing zpools the above message appears and I am unable to import zpools.

Issue Links

• is related to: LU-7991 Add project quota for ZFS
• is related to: LU-10016 Downgrading Lustre 2.10.1 ZFS to 2.9....

Comments

[Peter Jones]

Nathaniel

Can you please advise on this one?

Thanks

Peter

[Andreas Dilger]

This is pretty clearly the fault of the https://review.whamcloud.com/15294 patch, which is enabling user quota automatically at mount time:

static inline bool osd_dmu_userobj_accounting_available(struct osd_device *osd)
{
       if (unlikely(dmu_objset_userobjspace_upgradable(osd->od_os)))
               dmu_objset_userobjspace_upgrade(osd->od_os);

       return dmu_objset_userobjspace_present(osd->od_os);
}

This should instead just print a message at mount time like dnode accounting feature missing, run "zfs/zpool ..." to enable (prevents downgrade to ZFS < 0.7) or similar, and leave it up to the administrator to enable this feature. It should be up to them to enable this feature.

For our own testing, we should automatically enable this feature for newly-formatted ZFS datasets in libmount_utils_zfs.c.
In theory, the check in sanity-quota.sh test_7e should be changed from a ZFS version check to a check of the feature is enabled on the pool, but if we always format new ZFS pools with this feature then we should be OK.

I think this also needs to be included into Lustre 2.10.1 if at all possible, to avoid causing problems for users upgrading their system to use ZFS 0.7.

[Nathaniel Clark]

Looking at zpool-features(5):

       userobj_accounting

           GUID                   org.zfsonlinux:userobj_accounting
           READ-ONLY COMPATIBLE   yes
           DEPENDENCIES           extensible_dataset

           This feature allows administrators to account the object usage information by user and group.

           This  feature  becomes  active  as soon as it is enabled and will never return to being enabled.
           Each filesystem will be upgraded automatically when remounted, or when  new  files  are  created
           under  that filesystem.  The upgrade can also be started manually on filesystems by running `zfs
           set version=current <pool/fs>`. The upgrade process runs in the background and may take a  while
           to complete for filesystems containing a large number of files.

It implies that as soon as the old filesystem is mounted, the feature will be enabled and then the pool can't be downgraded. I'll test this.

[Nathaniel Clark]

It doesn't auto-upgrade; So I'll work up a patch in the vein Andreas recommended.

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28924
Subject: LU-9960 osd-zfs: don't auto-upgrade quota
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 20b43b86686696e02d5a8f549b412b8496a71e13

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28954
Subject: LU-9960 osd-zfs: don't auto-upgrade quota
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 22060779e670119293db38941ee318e747efc0fa

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/28954/
Subject: LU-9960 osd-zfs: don't auto-upgrade quota
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: c62476e202d7bd31e959b9682292319d20dd4dce

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28924/
Subject: LU-9960 osd-zfs: don't auto-upgrade quota
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bb6ed7dcde71b3878402f1ad4c2d88bc8edd8557

[Peter Jones]

Landed for 2.11