[LU-10048] osd-ldiskfs to truncate outside of main transaction

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.12.0, Lustre 2.10.4, Lustre 2.10.5
• Fix Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Major
• Reporter: Alex Zhuravlev
• Assignee: Alex Zhuravlev
• Resolution: Fixed

Description

this is needed to implement (transaction first; locking next) order to unify locking among MDT/OST/OUT

Issue Links

• is blocked by: LU-11764 osd_declare_qid() should not rely on ...
• is duplicated by: LU-11613 MDS and OSS locked up wait_transactio...
• is duplicated by: LU-8786 Terrible i/o performance of a test ap...
• is related to: LU-5152 Can't enforce block quota when unpriv...
• is related to: LU-8806 LFSCK hangs on MDT - osp_precreate_cl...
• is related to: LU-11685 removing file names and freeing inode...
• is related to: LU-11465 OSS/MDS deadlock in 2.10.5
• is related to: LU-5994 DT transaction start and object lock ...
• is related to: LU-13234 ldiskfs/namei.c:3310 ldiskfs_orphan_a...
• is related to: LU-12977 fix i_mutex for ldiskfs_truncate() in...

Comments

[Andreas Dilger]

The https://review.whamcloud.com/27488 patch is for ldiskfs, while the LU-8806 patch is for ZFS. Presumably they both need to be fixed?

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31293
Subject: LU-10048 ofd: take local locks within transaction
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a0ff9fb6374689e50674901f3ff3ede857d61848

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27488/
Subject: LU-10048 osd: async truncate
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cf29a5e7bfa254ccfcea023028fe7da80503c512

[Andreas Dilger]

Still one more patch to land.

[Lukasz Flis]

is there a backport for b2_10 available or planned?

Alex pointed this issue as duplicate of LU-11465  but we were not able to cherrypick changes in b2_10

We are experiencing MDT/OST lock-ups on 2_10_5 few times a day in the worst case

 

 

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33586
Subject: LU-10048 osd: async truncate
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 414b3ed9fd5aa975da26b8847e9cfe8a188b59ce

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33587
Subject: LU-10048 ofd: take local locks within transaction
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 7b6cdb405a519b11537316956affc4bfa4467f8d

[Mahmoud Hanafi]

Are the 2.10 ok to use? or do they still need additional work?

 

[Peter Jones]

Mahmoud

I would recommend holding off for now

Peter

[Andreas Dilger]

I think the https://review.whamcloud.com/33586 patch "LU-10048 osd: async truncate" is relatively safe - it has been in master for several months already, but I'm not sure it will fix the issue completely. Also, the patch https://review.whamcloud.com/33682
"Revert LU-5152 quota: enforce block quota for chgrp" may also help with MDT/OST lockups. It reverts a patch that was landed in 2.10.4 that introduced a circular dependency between servers in the quota handling.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31293/
Subject: LU-10048 ofd: take local locks within transaction
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9f79d4488fbb466647d1d09c2e6a1d3555d062fc

[Peter Jones]

So...is this ok to mark as resolved now?

[Peter Jones]

Andreas thinks yes

[Gerrit Updater]

Mark Roper ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37797
Subject: LU-10048 ofd: take local locks within transaction
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: afa505898952b9c094bd6fb4d9911032e4eefb68

[Aurelien Degremont (Inactive)]

We submitted a backport of this patch to b2_10 if anybody is looking for it.

@Lukasz, if you are still looking for it

 

[Gerrit Updater]

Etienne AUJAMES ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43277
Subject: LU-10048 ofd: take local locks within transaction
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 4dc5634dcdf3381a7ea698605a3c7a345f689a4c

[Etienne Aujames]

Hello,

I have backported the https://review.whamcloud.com/43277 ("LU-10048 ofd: take local locks within transaction") along with the https://review.whamcloud.com/43278 ("LU-13093 osd: fix osd_attr_set race").

It seems we trigger that bug in 2.12.6 with external journal (on flash dev) for rotational disk while several migrate thread is running on a robinhood client (512 thread).

jbd2 journal seems to be locked when the transaction is in T_LOCKED.

transaction_t.t_state = T_LOCKED
transaction_t.t_handle_count = 40
transaction_t.t_updates = 1
nbr of task in j_wait_transaction_locked = 324
nbr of task in j_wait_update = 0

[Alex Zhuravlev]

there is yet another patch under LU-10048 - LU-10048 osd: async truncate, cf29a5e7bfa254ccfcea023028fe7da80503c512 in master

[Alex Zhuravlev]

as for fast journal - we've got number of nodes running all the tests on RAM-backed devices 24h a day.

[Etienne Aujames]

The "LU-10048 osd: async truncate" has already landed on b2_12.

[Alex Zhuravlev]

please, generate full backtrace (

echo t >/proc/sysrq-trigger

) and attach to the ticket

[Etienne Aujames]

The issue seems to occurs relatively often with a lot of migrate threads (today 4 times on different OSTs).

We will test the backports quickly on a production environment (maybe this week).

I will try to get backtrace from the crashdump (manually triggered) tomorrow.

[Etienne Aujames]

I have added the our backtrace to this tickets: crash_bt_jbd2_locked_20210412.log

[Etienne Aujames]

Hello Alex,

Did you have the time to look to our backtrace?

If you need more data from the crashdump I can get you some.

[Etienne Aujames]

Hello,

We have applied the https://review.whamcloud.com/43277 ("LU-10048 ofd: take local locks within transaction") + https://review.whamcloud.com/43278 ("LU-13093 osd: fix osd_attr_set race") on the problematic filesystem. The issue never occurred after.

We were able to reproduce this issue in 5/10 min with many creations of small files and several threads doing file migrations ("lfs migrate" between OSTs).