[LU-7754] DNE3: osd-zfs gets into a livelock if transaction is too big

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Minor
• Reporter: Alex Zhuravlev
• Assignee: Alex Zhuravlev
• Resolution: Unresolved
• Labels: dne3, zfs
• Severity: 3

Description

ONLY=300k bash sanity.sh:

[ 89.828294] LNet: Service thread pid 4249 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[ 89.831356] Pid: 4249, comm: mdt01_001
[ 89.831895]
[ 89.831895] Call Trace:
[ 89.832451] [<ffffffff810ac6de>] ? getrawmonotonic+0x2e/0xc0
[ 89.833222] [<ffffffff810828c5>] __cond_resched+0x25/0x40
[ 89.834035] [<ffffffff814e521a>] _cond_resched+0x2a/0x40
[ 89.834810] [<ffffffff814e5f11>] mutex_lock+0x11/0x40
[ 89.835565] [<ffffffffa07741b4>] dmu_tx_assign+0x284/0x500 [zfs]
[ 89.836410] [<ffffffffa0a1ae32>] osd_trans_start+0xb2/0x410 [osd_zfs]
[ 89.837282] [<ffffffffa032de15>] top_trans_start+0x255/0x9c0 [ptlrpc]
[ 89.838090] [<ffffffffa0bb88f9>] lod_trans_start+0x59/0x60 [lod]
[ 89.838854] [<ffffffffa0ae3cdf>] mdd_trans_start+0xf/0x20 [mdd]
[ 89.839594] [<ffffffffa0acf1a0>] mdd_create+0x1170/0x1c70 [mdd]

Issue Links

• is related to: LU-9096 sanity test_253: File creation failed...
• is related to: LU-8267 sanity test_129: (osd_internal.h:1137...

Comments

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18341
Subject: LU-7754 osd: osd-zfs should not wait indefinitely for a TXG
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ebef50ed6f5032697f05c4bcc20c7fb329423a17

[Andreas Dilger]

Your patch turn this from a hang into a failure. That is an improvement, but it doesn't explain why this test failed? Do you have an unusual config (small MDT?) or is there some regression that makes the transaction too large?

[Alex Zhuravlev]

sanity/300k tries to create a big striped directory:

$LFS setdirstripe -i 0 -c512 $DIR/$tdir/striped_dir

with default MDSSIZE=200000 DMU fails to start such a big transaction.

[Andreas Dilger]

How large is the transaction? Do we have a larger MDS size in our testing?

I guess this is because we don't run DNE + ZFS by default.

[Alex Zhuravlev]

transaction calculations:
mem 4986830848, asize 119683940352, fsize 8506441728, usize 8497152000

it seem to fail because of insufficient memory: 4986830848 (4755MB) is needed while the test system had 4GB in total.

[Andreas Dilger]

That is 4755 MB / 512 stripes = 9 MB/stripe which seems like a lot of space to reserve? I thought we got away from O(n^2) transaction sizes for striped directories?