[LU-2435] inode accounting in osd-zfs is racy

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.4.1
• Labels: RZ_LS llnl quota
• Severity: 3

Epic

server

• Assignee: Jinshan Xiong (Inactive)
• Reporter: Brian Behlendorf

Description

Observed on the MDT during weekend testing with an mixed I/O workload. I agree failure here shouldn't be fatal, however I'm not sure we should even bother telling the administrator. If it's true there's nothing that needs to be (or can be) done, then there's no point is logging this to the console. Changing it to a CDEBUG() would be preferable. If however the administrator needs to do something (which I don't believe is the case) that should be part of the message.

2012-05-04 17:12:17 LustreError: 14242:0:(osd_handler.c:997:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000023a1:0xf3ed:0x0] from accounting ZAP for grp 36427 (-2)
2012-05-04 17:12:17 LustreError: 14242:0:(osd_handler.c:997:osd_object_destroy()) Skipped 1 previous similar message
2012-05-04 17:16:22 LustreError: 14244:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x20000239a:0x991a:0x0] from accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:30 LustreError: 5618:0:(osd_handler.c:2003:osd_object_create()) lcz-MDT0000: failed to add [0x20000239a:0x9d46:0x0] to accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:34 LustreError: 14241:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000013ca:0x1952b:0x0] from accounting ZAP for usr 36427 (-2)
2012-05-04 17:16:53 LustreError: 6778:0:(osd_handler.c:997:osd_object_destroy()) lcz-MDT0000: failed to remove [0x2000013ca:0x19e3f:0x0] from accounting ZAP for grp 36427 (-2)
2012-05-04 17:31:21 LustreError: 6778:0:(osd_handler.c:2003:osd_object_create()) lcz-MDT0000: failed to add [0x200002393:0x1ee18:0x0] to accounting ZAP for usr 36427 (-2)
2012-05-04 17:31:32 LustreError: 6770:0:(osd_handler.c:991:osd_object_destroy()) lcz-MDT0000: failed to remove [0x200002395:0x167e0:0x0] from accounting ZAP for usr 36427 (-2)

Issue Links

• is blocking: LU-7895 zfs metadata performance improvements
• is related to: LU-8326 sanity-quota test_21 test failed to respond and timed out
• is related to: LU-6965 osd_object.c:928:osd_attr_set
• is related to: LU-8927 osp-syn processes contending for osq_lock drives system cpu usage > 80%
• is related to: LU-9192 lfs quota reports wrong file count when using ZFS backend
• is related to: LU-2619 Bogus value of dqb_curinodes returned by osc_quotactl
• is related to: LU-5638 sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn't 0. 1
• is related to: LU-7991 Add project quota for ZFS
• is related to: LU-7991 Add project quota for ZFS

Comments

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/15294/
Subject: LU-2435 osd-zfs: use zfs native dnode accounting
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 11afef00b6af407b8987076bd4f1ec9bc77eb75e

[Gerrit Updater]

Jinshan Xiong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26090
Subject: LU-2435 osd-zfs: for test only
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eac70d59649d2ab3bd5b7d0acc94cbf67a430251

[Alex Zhuravlev]

depends on how exactly we call dmu_tx_hold_zap(): the more specific, the less credits. unfortunately when the key is specified, then declaration becomes very expensing due to hash lock/lookup. also, the final calculation depends on inflation size:
asize = spa_get_asize(tx->tx_pool->dp_spa, towrite + tooverwrite);

spa_get_asize(spa_t *spa, uint64_t lsize)
{
return (lsize * spa_asize_inflation);

int spa_asize_inflation = 24;

[Jinshan Xiong (Inactive)]

Is ~0.5MB the worst case or on average? Can you please describe this in detail?

[Alex Zhuravlev]

each ZAP declaration adds ~0.5MB to space/memory reservation. so dnode accounting adds ~1MB, which is 20% of single-stripe object creation.

[Andreas Dilger]

The latest version of the patch is actually https://github.com/zfsonlinux/zfs/pull/3983 and not 3723.

[Jinshan Xiong (Inactive)]

I will work on this.

[Peter Jones]

Jinshan

The link #15180 does not seem to work. Could you please confirm whether that is correct?

Peter

[Jinshan Xiong (Inactive)]

The proposal made by Andreas worked. This is what I have done:

0. prepare an 'old' lustre zfs backend and copy some files into the file system with user 'tstusr';
1. apply patch http://review.whamcloud.com/15180 to zfs repo and patch 15294 to lustre;
2. recompile and install zfs and lustre;
3. upgrade pool by 'zpool upgrade lustre-mdt1';
4. mount Lustre;
5. upgrade mdt to use native dnode accounting by 'lctl set_param osd-zfs.lustre-MDT0000.quota_native_dnused_upgrade';
6. and it worked. I can get correct dnode use accounting by 'lfs quota -u tstusr /mnt/lustre';