[#LU-50] pdirops patch for ldiskfs

[LU-50] pdirops patch for ldiskfs Created: 23/Jan/11 Updated: 28/Apr/20 Resolved: 22/Jun/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.1.0
Fix Version/s:	Lustre 2.2.0

Type:

New Feature

Priority:

Major

Reporter:

Liang Zhen (Inactive)

Assignee:

Liang Zhen (Inactive)

Resolution:

Fixed

Votes:

Labels:

None

Attachments:

OpenSFS_Implementation_Milestone.pdf

OpenSFS_PDirOps_Demonstration_Milestone.pdf

OpenSFS_PDirOps_HighLevelDesign.pdf

OpenSFS_PDirOps_Scope_Statement.pdf

Opensfs_PDO_performance.pdf

Parallel Directory Solution Architecture - OpenSFS - Whamcloud Community.pdf

mdtest_lustre.patch

Issue Links:

Related
is related to	~~LU-13054~~	MDS kernel BUG at ldiskfs/htree_lock....	Resolved
is related to	~~LU-1365~~	Implement ldiskfs LARGEDIR support fo...	Resolved
is related to	~~LU-896~~	change e2fsprogs to make it allow dir...	Resolved

Rank (Obsolete):

4800

Description

Changing operations of shared directory are not good even with all SMP improvements, I made some efforts to localize changing operations on a few cores to decrease contention and they can somehow help to improvement performance of pdirops (see my comment on ~~LU-55~~).

At the same time, it's worth to try to increase pdirops performance in ldiskfs level. We had a dynlock based pdirops patch for ldiskfs several years ago, but we didn't maintain it. Actually, I found it's hard to maintain it because it's a complex patch and made a lot of changes to htree code of ext3/ext4. So I'm thinking about to have a more simpler implementation of pdirops patch.

The idea is quite simple (just like how we implement pdirop in ldlm level):

design & implement a advanced lock with multiple modes (CR, PR, CW, PW, EX)
it's just a resource lock resides in osd-ldiskfs & ldiskfs, which will replace current osd_object::oo_ext_idx_sem
we can partially protect resource by locking N hash keys after holding CR & CW
- for CR, sub locking is PR
- for CW, sub locking is PW
for !is_dx(dir), change operations are always holding EX lock on dir, lookup & readdir hold PR lock on dir
for is_dx(dir)
- change operations take CW lock, then take two sub-locks: name-hash as the first key, block number as the second key
- lookup take CR lock, then take two sub-locks: name-hash as the first key, block number as the second key
- readdir take PR lock (it's not necessary to take PR lock, but we already have PR lock in ldlm level, so it doesn't matter)
if we need to split any blocks (name entries block or indexing block) while have CW lock, we release CW lock and take EX lock and retry.
we can disable pdirops ldiskfs by turning osd-ldiskfs pdir_ldiskfs=0/1, if we disabled pdirops for osd-ldiskfs then osd_ldiskfs always EX lock dir on changing and PR lock dir on lookup/readdir.

Although implementation of the lock is a little complex, but we just need very few changes to ext4/ldiskfs and osd-ldiskfs with this lock:

osd-ldiskfs: replace oo_ext_idx_sem with the new lock
ldiskfs: switch lock mode if we need to split/append (in ldiskfs).

PS: I'm thinking about rename parl_lock to htree_lock, because it's really designed for htree operations, and Eric hates confusing naming like "parl_lock"

Comments

Comment by Alex Zhuravlev [ 24/Jan/11 ]

> if we need to split any blocks (name entries block or indexing block) while have CW lock, we release CW lock and take EX lock and retry.

sounds like you're going to change APIs between MDT through ldiskfs?

Comment by Liang Zhen (Inactive) [ 24/Jan/11 ]

no, all changes are just inside osd-ldiskfs and ldiskfs
osd-ldiskfs changes are simple:
just replace oo_ext_idx_sem with the new lock and some small changes.

ldiskfs changes are simple as well... just need to change lock mode if necessary.
The only complex thing in the patch is implementation of the lock, it has nothing to do with current ldlm although the idea is from ldlm.

Comment by Liang Zhen (Inactive) [ 24/Jan/11 ]

the patch is here: http://review.whamcloud.com/#change,210

Comment by Alex Zhuravlev [ 24/Jan/11 ]

then it's not clear how MDT knows when to release CW and get EX, how ldiskfs knows when to interrupt split and when do no.

Comment by Liang Zhen (Inactive) [ 24/Jan/11 ]

it's a totally osd/diskfs level lock so MDT doesn't need to know anything, but ldiskfs should know when needs to split/append(we patched it), whenever ldiskfs needs to change the htree, it retry with EX lock.

Comment by nasf (Inactive) [ 24/Jan/11 ]

For split case, whether we can upgrade from CW lock to EX lock directly without releasing CW lock firstly?

Comment by Liang Zhen (Inactive) [ 24/Jan/11 ]

In the patch, I defined function: parl_change_lock, which means we can atomically release lock and gain lock with new mode if there is no conflict, otherwise it will be blocked.

Comment by Liang Zhen (Inactive) [ 22/Feb/11 ]

some interesting numbers (single node, 4-cores * 2-hyper-threading, ramdisk MDT), 16 mounts & 16 threads opencreate 128K files under shared directory

(fname len) 200 100 40
ldiskfs : 4400 11472 11808
pdirop : 6976 15994 18192
iam-dir : 10496 16816 18128

as we can see, both pdirop patch and iam-dir can help on performance, but pdirop patch can't help too much if filename is extremely long (200 character), it's expected because we always take exclusive lock on directory if we need to split any block.

Regards
Liang

Comment by Andreas Dilger [ 22/Feb/11 ]

The site http://www.pdsi-scidac.org/fsstats/ has interesting information about filesystems, including average filename lengths (though the data is getting a bit old by now). It shows that over 99% of most filesystems have filenames are 40 characters or less, and virtually all filenames are 100 characters or less.

That means I'm not terribly worried about the 200-character performance difference. That is doubly true if this test was done on a filesystem with 1024-byte blocksize, which could only have 4 filenames per block before splitting.

Comment by Liang Zhen (Inactive) [ 22/Feb/11 ]

Andreas, these are tests over lustre client, so I'm using 4K blocksize for all cases.
I actually suspect it's not about split, when I set namelen to 190+ pdirop performance is still good (13K+/sec), but if I set namelen to 200+ I saw a big drop of performance (7K/sec) and I don't know why.
IAM-dir doesn't have such an issue.

Liang

Comment by Liang Zhen (Inactive) [ 29/Mar/11 ]

I've made a lot of changes to previous patch:

of course, rename, parl_lock* are changed to htree_lock*
after some tests, I found lock-htree-on-split is not good enough for performance because:
- split happens more frequently than I expected
- split takes much longer time than a regular name entry insertion, so we exclusively lock the htree very often.
the new locking protocol is:
- lock the leaf block (DE block) on inserting/deleting
- lock the last level index block (DX block) on splitting of leaf block (DE block)
- lock the whole tree on splitting of any index block(DX block), the chance of doing this is almost likely less than 0.1%
I've got really good testing results with this patch
downside of these improvements:
- patch is bigger
- htree_lock is more complex
upside of the new patch:
- patch is still clean and easy to maintain, 70% are just new code for implementation of htree_lock
- better performance than previous patch

Comment by Liang Zhen (Inactive) [ 29/Mar/11 ]

change it to 2.1

Comment by Build Master (Inactive) [ 29/Mar/11 ]

Integrated in reviews-centos5 #609
~~LU-50~~ pdirops patch for ldiskfs

Liang Zhen : c4ab2660caf219a125a91bb33fb5a48c384f2c36
Files :

ldiskfs/ldiskfs/autoMakefile.am
ldiskfs/kernel_patches/patches/ext4_pdirop.patch
lustre/osd-ldiskfs/osd_handler.c
ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series
lustre/osd-ldiskfs/osd_internal.h
ldiskfs/ldiskfs/Makefile.in

Comment by James A Simmons [ 03/Oct/11 ]

I have seen some other tickets implementing this feature. Now that Lustre 2.2 is in the works we can revive this work.

Comment by Andreas Dilger [ 09/Dec/11 ]

There need to be some follow-on bugs filed and fixed for this code in order of priority:

test LARGEDIR and 3-level htree for local ldiskfs in conf-sanity.sh with 1kB blocksize up to 100k entries with 255-byte names (3-level exceeded at 48k entries)
test LARGEDIR and >2GB directories with Lustre in parallel-scale using 255-byte names and 10M entries (2GB exceeded at 4M entries, 4GB exceeded at 8M entries)
e2fsprogs support needs to be added for 3-level htree
e2fsprogs support for directories larger than 2GB
e2fsprogs test for 3-level htree e2fsck, corruptions
e2fsprogs add LARGEDIR feature to "tests/f_random_corruption"
once e2fsprogs is released and available for download, a patch is needed to enable large_dir on new filesystems with mkfs.lustre (probably for 2.3 since I don't think the maximum directory size is a limit for most users today)
updates to the user manual and release notes to describe how to enable the LARGEDIR feature with tune2fs
separate the INCOMPAT_LARGEDIR features (>2GB, 3-level htree) patches from the htree-lock patch so that they can be submitted upstream