Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14139

batched statahead processing

Details

    • New Feature
    • Resolution: Fixed
    • Minor
    • Lustre 2.16.0
    • None
    • None
    • 9223372036854775807

    Description

      Batched metadata processing may get a big performance boost.
      Batched statahead can also increase the performance for directory listing operations such as ls.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-16365 cached 'ls -l' is slow

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17783 Wrong buffer for field 'batch_update_reply' in format 'MDS_BATCH' from statahead for old server

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-8582 Interop: master<->b2_8 - sanity test_255a: test failed to respond and timed out

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-10334 Ubuntu1604 client sanity-103a: FAIL: ACL misc test failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-14868 sanity: all subtests pass but test suite fails with TIMEOUT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-16247 do open()+readahead for DoM files as batched operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-16355 batch dirty buffered write of small files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-23 Improve performance for traversing large directory with readdir+

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Technical task - A technical task. LU-10280 DoM cross-file open+readahead via statahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14380 Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14138 Move more members in PTLRPC request into pill

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14361 Add support for statahead pattern with regularized file name

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LU-15975 Statahead_V1 Features

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-16139 statahead: avoid new RPC and long wait in the statahead interpret callback

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-10938 Metadata writeback cache support

          • Major - Major loss of function.
          • Open

          Technical task - A technical task. LU-13045 WBC2: batch metadata update

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-16096 recovery: handle compatibility during upgrade for new replay data format

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14361 Add support for statahead pattern with regularized file name

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14393 Batching Processing Framework for Lustre

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-15550 WBC: retry the batched RPC when the reply buffer is overflowed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-15562 statahead: using try lock for batched statahead to avoid deadlock with rename and migrate operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-14139] batched statahead processing
            pjones Peter Jones added a comment -

            Landed for 2.16

            pjones Peter Jones added a comment - Landed for 2.16
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41220/
            Subject: LU-14139 statahead: add test for batch statahead processing
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: cd1e7bdd903ada4467064eaf8613ec62a358fa1c

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41220/ Subject: LU-14139 statahead: add test for batch statahead processing Project: fs/lustre-release Branch: master Current Patch Set: Commit: cd1e7bdd903ada4467064eaf8613ec62a358fa1c
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/43707/
            Subject: LU-14139 ptlrpc: grow PtlRPC properly when prepare sub request
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 5a2dfd36f9c2b6c10ab7ba44b0e9e86372623fde

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/43707/ Subject: LU-14139 ptlrpc: grow PtlRPC properly when prepare sub request Project: fs/lustre-release Branch: master Current Patch Set: Commit: 5a2dfd36f9c2b6c10ab7ba44b0e9e86372623fde
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40945/
            Subject: LU-14139 ptlrpc: grow reply buffer properly for batch request
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 5edb883b44ac707528ce2c0bc812d65b9ffb4a50

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40945/ Subject: LU-14139 ptlrpc: grow reply buffer properly for batch request Project: fs/lustre-release Branch: master Current Patch Set: Commit: 5edb883b44ac707528ce2c0bc812d65b9ffb4a50
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40943/
            Subject: LU-14139 statahead: add stats for batch RPC requests
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: a20f25d24b5f0ce7b5e77f7c596bffd0450cbdae

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40943/ Subject: LU-14139 statahead: add stats for batch RPC requests Project: fs/lustre-release Branch: master Current Patch Set: Commit: a20f25d24b5f0ce7b5e77f7c596bffd0450cbdae
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40720/
            Subject: LU-14139 statahead: batched statahead processing
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 4435d0121f72aac3ad01c98a33b265a496359890

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40720/ Subject: LU-14139 statahead: batched statahead processing Project: fs/lustre-release Branch: master Current Patch Set: Commit: 4435d0121f72aac3ad01c98a33b265a496359890
            adilger Andreas Dilger added a comment -

            Is there a patch that converts ldlm to use rhashtable?

            adilger Andreas Dilger added a comment - Is there a patch that converts ldlm to use rhashtable?
            qian_wc Qian Yingjin added a comment - - edited

            From the attaching FG, lots of time costs are:
            __ldlm_handle2lock () ->class_handle2object()
            ldlm_resource_get() >cfs_hash_bd_lookup_intent()>ldlm_res_hop_keycmp()

            Each of them costs 5% for data and metadata DLM lock matching on the client side. All takes about 20% of the total sample...
            It seems that the hash for looking up the lock handle and resource take lots of time.
            And ldlm_res_hop_keycmp() reaches 5%, it means there are lots of elements in the same bucket of the hash table. We should increase the hash table and use the resizable hash table.

            For class_handle2object for locks, we should maintain the lock handle per target (osc/mdc) lock namespace, not shared a single global hash table for handle on the client or even the server.

            qian_wc Qian Yingjin added a comment - - edited From the attaching FG, lots of time costs are: __ldlm_handle2lock () ->class_handle2object() ldlm_resource_get() >cfs_hash_bd_lookup_intent() >ldlm_res_hop_keycmp() Each of them costs 5% for data and metadata DLM lock matching on the client side. All takes about 20% of the total sample... It seems that the hash for looking up the lock handle and resource take lots of time. And ldlm_res_hop_keycmp() reaches 5%, it means there are lots of elements in the same bucket of the hash table. We should increase the hash table and use the resizable hash table. For class_handle2object for locks, we should maintain the lock handle per target (osc/mdc) lock namespace, not shared a single global hash table for handle on the client or even the server.
            sihara Shuichi Ihara added a comment -

            there are no LNET messages go out in 2nd 'ls -l'.
            I've opened a separate ticket for more investigations. https://jira.whamcloud.com/browse/LU-16365

            sihara Shuichi Ihara added a comment - there are no LNET messages go out in 2nd 'ls -l'. I've opened a separate ticket for more investigations. https://jira.whamcloud.com/browse/LU-16365
            sihara Shuichi Ihara added a comment -

            it might be not related to LU-14139 direclty, but still interested in why Lustre still takes 25sec for second 'ls -l'.

            It looks like the client is still sending 125k x 8=1M glimpse RPCs to the OSTs to fetch the file size. I guess this is because the client is not caching all 1M of the OST DLM locks on the objects between "ls -l" calls? Did you check OSC LRU size? I suspect this would have a sudden jump in speed once the number of files is small enough that all the OST DLM locks are cached. 

            As far as I observed, all locks seems to be kept as well. nothing shrinks or re-granting locks between two 'ls -l'.

            [root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size
ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=0

[sihara@ec01 ~]$ time ls -l /exafs/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null

real	0m27.249s
user	0m9.111s
sys	0m13.544s

[root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size
ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=2
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=4
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=1000004
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=125089
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=125073
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=125091
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=125085
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=125295
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=125285
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=125103
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=125099
ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=2
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=4
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=1000004
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=125089
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=125073
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=125091
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=125085
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=125295
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=125285
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=125103
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=125099

[sihara@ec01 ~]$ time ls -l /exafs/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null

real	0m25.615s
user	0m8.982s
sys	0m16.588s

[root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size
ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=2
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=4
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=1000004
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=125089
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=125073
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=125091
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=125085
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=125295
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=125285
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=125103
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=125099
ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200
ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=2
ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=4
ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=1000004
ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0
ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=125089
ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=125073
ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=125091
ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=125085
ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=125295
ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=125285
ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=125103
ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=125099

            Here is same 1M files tests for ext4 on local disk of client.

            [root@ec01 ~]# echo 3 > /proc/sys/vm/drop_caches
[sihara@ec01 ~]$ time ls -l /tmp/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/  > /dev/null

real	0m16.999s
user	0m8.956s
sys	0m5.855s
[sihara@ec01 ~]$ time ls -l /tmp/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/  > /dev/null

real	0m11.832s
user	0m8.765s
sys	0m3.051s

            I'm also attaching FG which captured at second 'ls -l' on Lustre.

            sihara Shuichi Ihara added a comment - it might be not related to LU-14139 direclty, but still interested in why Lustre still takes 25sec for second 'ls -l'. It looks like the client is still sending 125k x 8=1M glimpse RPCs to the OSTs to fetch the file size. I guess this is because the client is not caching all 1M of the OST DLM locks on the objects between "ls -l" calls? Did you check OSC LRU size? I suspect this would have a sudden jump in speed once the number of files is small enough that all the OST DLM locks are cached.  As far as I observed, all locks seems to be kept as well. nothing shrinks or re-granting locks between two 'ls -l'. [root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=0 [sihara@ec01 ~]$ time ls -l /exafs/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null real 0m27.249s user 0m9.111s sys 0m13.544s [root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=2 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=4 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=1000004 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=125089 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=125073 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=125091 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=125085 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=125295 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=125285 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=125103 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=125099 ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=2 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=4 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=1000004 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=125089 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=125073 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=125091 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=125085 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=125295 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=125285 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=125103 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=125099 [sihara@ec01 ~]$ time ls -l /exafs/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null real 0m25.615s user 0m8.982s sys 0m16.588s [root@ec01 ~]# lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.*.lru_size ldlm.namespaces.MGC10.0.11.208@o2ib12.lock_count=4 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lock_count=2 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lock_count=4 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lock_count=1000004 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lock_count=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lock_count=125089 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lock_count=125073 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lock_count=125091 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lock_count=125085 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lock_count=125295 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lock_count=125285 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lock_count=125103 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lock_count=125099 ldlm.namespaces.MGC10.0.11.208@o2ib12.lru_size=3200 ldlm.namespaces.exafs-MDT0000-mdc-ffff9b96b8157800.lru_size=2 ldlm.namespaces.exafs-MDT0001-mdc-ffff9b96b8157800.lru_size=4 ldlm.namespaces.exafs-MDT0002-mdc-ffff9b96b8157800.lru_size=1000004 ldlm.namespaces.exafs-MDT0003-mdc-ffff9b96b8157800.lru_size=0 ldlm.namespaces.exafs-OST0000-osc-ffff9b96b8157800.lru_size=125089 ldlm.namespaces.exafs-OST0001-osc-ffff9b96b8157800.lru_size=125073 ldlm.namespaces.exafs-OST0002-osc-ffff9b96b8157800.lru_size=125091 ldlm.namespaces.exafs-OST0003-osc-ffff9b96b8157800.lru_size=125085 ldlm.namespaces.exafs-OST0004-osc-ffff9b96b8157800.lru_size=125295 ldlm.namespaces.exafs-OST0005-osc-ffff9b96b8157800.lru_size=125285 ldlm.namespaces.exafs-OST0006-osc-ffff9b96b8157800.lru_size=125103 ldlm.namespaces.exafs-OST0007-osc-ffff9b96b8157800.lru_size=125099 Here is same 1M files tests for ext4 on local disk of client. [root@ec01 ~]# echo 3 > /proc/sys/vm/drop_caches [sihara@ec01 ~]$ time ls -l /tmp/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null real 0m16.999s user 0m8.956s sys 0m5.855s [sihara@ec01 ~]$ time ls -l /tmp/testdir/mdtest.out/test-dir.0-0/mdtest_tree.0/ > /dev/null real 0m11.832s user 0m8.765s sys 0m3.051s I'm also attaching FG which captured at second 'ls -l' on Lustre.

            People

              qian_wc Qian Yingjin
              qian_wc Qian Yingjin
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: