[LU-2435] inode accounting in osd-zfs is racy

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.4.1
• Labels: RZ_LS llnl quota
• Severity: 3

Epic

server

• Assignee: Jinshan Xiong (Inactive)
• Reporter: Brian Behlendorf

Description

Observed on the MDT during weekend testing with an mixed I/O workload. I agree failure here shouldn't be fatal, however I'm not sure we should even bother telling the administrator. If it's true there's nothing that needs to be (or can be) done, then there's no point is logging this to the console. Changing it to a CDEBUG() would be preferable. If however the administrator needs to do something (which I don't believe is the case) that should be part of the message.

2012-05-04 17:12:17 LustreError: 14242:0:(osd_handler.c:997:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000023a1:0xf3ed:0x0] from accounting ZAP for grp 36427 (-2)
2012-05-04 17:12:17 LustreError: 14242:0:(osd_handler.c:997:osd_object_destroy()) Skipped 1 previous similar message
2012-05-04 17:16:22 LustreError: 14244:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x20000239a:0x991a:0x0] from accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:30 LustreError: 5618:0:(osd_handler.c:2003:osd_object_create()) lcz-MDT0000: failed to add [0x20000239a:0x9d46:0x0] to accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:34 LustreError: 14241:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000013ca:0x1952b:0x0] from accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:53 LustreError: 6778:0:(osd_handler.c:997:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000013ca:0x19e3f:0x0] from accounting ZAP for grp 36427 (-2)
2012-05-04 17:31:21 LustreError: 6778:0:(osd_handler.c:2003:osd_object_create()) lcz-MDT0000: failed to add [0x200002393:0x1ee18:0x0] to accounting ZAP for usr 36427 (-2)
2012-05-04 17:31:32 LustreError: 6770:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x200002395:0x167e0:0x0] from accounting ZAP for usr 36427 (-2)

Issue Links

• is blocking: LU-7895 zfs metadata performance improvements
• is related to: LU-8326 sanity-quota test_21 test failed to respond and timed out
• is related to: LU-6965 osd_object.c:928:osd_attr_set
• is related to: LU-8927 osp-syn processes contending for osq_lock drives system cpu usage > 80%
• is related to: LU-9192 lfs quota reports wrong file count when using ZFS backend
• is related to: LU-2619 Bogus value of dqb_curinodes returned by osc_quotactl
• is related to: LU-5638 sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn't 0. 1
• is related to: LU-7991 Add project quota for ZFS
• is related to: LU-7991 Add project quota for ZFS

Comments

[Alex Zhuravlev]

depends on how exactly we call dmu_tx_hold_zap(): the more specific, the less credits. unfortunately when the key is specified, then declaration becomes very expensing due to hash lock/lookup. also, the final calculation depends on inflation size:
asize = spa_get_asize(tx->tx_pool->dp_spa, towrite + tooverwrite);

spa_get_asize(spa_t *spa, uint64_t lsize)
{
return (lsize * spa_asize_inflation);

int spa_asize_inflation = 24;

[Jinshan Xiong (Inactive)]

Is ~0.5MB the worst case or on average? Can you please describe this in detail?

[Alex Zhuravlev]

each ZAP declaration adds ~0.5MB to space/memory reservation. so dnode accounting adds ~1MB, which is 20% of single-stripe object creation.

[Andreas Dilger]

The latest version of the patch is actually https://github.com/zfsonlinux/zfs/pull/3983 and not 3723.

[Jinshan Xiong (Inactive)]

I will work on this.

[Peter Jones]

Jinshan

The link #15180 does not seem to work. Could you please confirm whether that is correct?

Peter

[Jinshan Xiong (Inactive)]

The proposal made by Andreas worked. This is what I have done:

0. prepare an 'old' lustre zfs backend and copy some files into the file system with user 'tstusr';
1. apply patch http://review.whamcloud.com/15180 to zfs repo and patch 15294 to lustre;
2. recompile and install zfs and lustre;
3. upgrade pool by 'zpool upgrade lustre-mdt1';
4. mount Lustre;
5. upgrade mdt to use native dnode accounting by 'lctl set_param osd-zfs.lustre-MDT0000.quota_native_dnused_upgrade';
6. and it worked. I can get correct dnode use accounting by 'lfs quota -u tstusr /mnt/lustre';

[Brian Behlendorf]

To my knowledge there is no existing code for dnode accounting in ZFS. How to implement it in a reasonable way was described above but no one has yet done that work. However once it is done and merged we'll have to maintain it forever since old filesystems may always need to be upgraded.

[Andreas Dilger]

There is already existing code to do the dnode iteration to enable user/group block quota on an existing filesystem, and I suspect that this is still part of the ZFS code, and may never be deleted. There hopefully shouldn't be the need for much new code beyond the existing block quota iterator.

[Jinshan Xiong (Inactive)]

I'm going to start the 2nd phase of this work. At the time of dnode accounting feature is enabled, it will check if the file system is a legacy one. In that case, it will launch a thread to iterate all dnodes in all dataset and repair dnode accounting. I have two concerns for this task:

1. this piece of code will become dead once the scanning task is done, and new FS won't need it at all. Will ZFS upstream team accept this code? Based on the same concern, I will make an independent patch based on https://github.com/zfsonlinux/zfs/pull/2577;
2. does it need to do anything special for snapshot and clone?

Thanks.

[Jinshan Xiong (Inactive)]

I pushed patch 15294 to gerrit to use ZFS dnode accounting for Lustre inode accounting. This is the work phase one. The second phase is to launch a thread to iterate all dnodes in the objset of MDT to repair accounting of legacy file system.