https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-56 Lustre <p>Please post the SMP patches to bugzilla and then push the changes to a branch on review.whamcloud.com/lustre. Keep the branch current as changes are made to the patches during the inspection/landing process, because we'll also be testing this branch at LLNL and possibly other places.</p> LU-56

Finish SMP scalability work

Task Major Resolved Fixed Liang Zhen Robert Read Fri, 24 Sep 2010 13:51:22 +0000 Thu, 13 May 2021 18:56:06 +0000 Fri, 28 Sep 2012 11:41:42 +0000 Lustre 2.3.0 Lustre 2.3.0 0 17 <p>I've merged NUMA code into SMP patches of LNet and ptlrpc service in last week, will start to post patches soon.</p> <p>I'm pushing changes into liang/b_hd_smp, those patches can be landed in weeks will not be pushed in.<br/> Also, I'm working on another three issues:</p> <ol> <li>move lprocfs_counter to libcfs <ul> <li>current lprocfs_counter is not very efficient and has a small defect</li> <li>we want to use percpu counter inside libcfs and LNet</li> </ul> </li> <li>general purpose threads pool APIs <ul> <li>several modules of Lustre have their own implementation to create/destroy thread pool, if we have a general purpose thread pool implementation, we can get ride of some code redundance.</li> </ul> </li> <li>MDD threads pool for create/link/unlink <ul> <li>As we know, on large scale SMP system, SMP scalability of diskfs will be bottleneck of these operations under shared directory, there is no easy solution of this.</li> <li>I had a discussion with Oleg last week, we got agreement about design MDD threads pool like this way: <ul> <li>MDD has a threads pool, all threads are CPU affinity</li> <li>MDD also has two levels caches <ul> <li>percpu fid cache<br/> parent fid of create/unlink/link will be added to top local CPU cache, if it's already in this cache, it just pop it up and increase refcount on the parent_fid, the fid can be discarded because cache has limited size (i.e: 8)</li> <li>global fid cache<br/> if refcount on the fid is larger than a value (i.e: 128), we also enqueue the fid into a global cache, if we found the fid is already in the global cache, we just increase refcount on the global cached fid. When refcount on global cached fid is larger than a specified value N(i.e: N=4), which means, there are "N" cores are create/link/unlink a lot of files under the same directory, we allocate 1 or 2 cores for this parent fid, all FS operations like create/link/unlink will be dispatched to MDD thread on these 1 or 2 cores.</li> </ul> </li> <li>if the parent fid is not in global cache, or global refcount is lower than specified value, we still process FS operations on thread context of caller</li> <li>we can easily enable/disable the caches for one or all operations.</li> <li>cache size is also configurable</li> </ul> </li> </ul> </li> </ol> <p>Liang</p> <p>Liang, I don't quite understand what you mean about "patches can be landed in weeks will not be pushed in." We need a local branch with all the SMP patches in it so we can start testing this at LLNL. Can we use liang/b_hd_smp for this?</p> <p>Robert, Yujian &amp; Vitaly are in progress of testing &amp; landing some patches, I think they can be done in weeks, so I think probably we don't have to push them into b_hd_smp because they will be in main branch very soon. </p> <p>All patches are already in branch liang/b_hd_smp, prototype of mdd thread pool + dir cache are already in the branch.<br/> Jim is going to test with it.</p> <p>made some improvements to mdd dir cache + transaction threads pool, I will post the whole idea at here later.</p> <p>I've created another ticket to track improvements on la_get (<a href="https://jira.whamcloud.com/browse/LU-36" title="mdd_la_get is called too many times in MDD" class="issue-link" data-issue-key="LU-36"><del>LU-36</del></a>)</p> <p>Description of MDD dirops cache &amp; transaction schedulers</p> <ul> <li>each CPU has a local fid cache (it's a 8 buckets hash-table, each bucket is a 8 items LRU, so we can cache 64 fids by default, of course we can increase number of buckets)</li> <li>transaction schedulers are percpu threads pool in MDD layer</li> <li>each time we create/remove file from a directory, we push the fid into the local CPU fid cache(LRU)</li> <li>if the fid is already in the cache, then just increase a refcount(# of hits) on the fid and move the fid to top of LRU</li> <li>when we push a new fid into local cache, the fid at bottom of LRU will be popped out</li> <li>if refcount(# of hits) of fid in local CPU cache &gt;= N (default is 32), we try to lookup the fid in global fid cache, which is a hash-table as well (do nothing if it already has reference to fid in global cache). <ul> <li>if the fid is not in the global cache, we just add it to global cache and set refcount to 1 (1 CPU is using it)</li> <li>if the fid is already in global cache, we increase refcount of fid in global cache</li> </ul> </li> <li>if refcount of fid in global cache &lt; M, we just return -1 and continue to run transaction in context of current thread (mdt service thread)</li> <li>if refcount of fid in global cache is &gt;= M (M is number of CPUs, which can be 2, 4... as many as we can get good performance of shared directory with ldiskfs/ext4, it's tunable), we specify M CPUs for the fid and return one CPU id within those M CPUs, if caller get a CPU id different with current CPU id, it will launch a MDD level "transaction request", and wakeup one MDD transaction thread to handle the "transaction request"</li> </ul> <p>By this way, we only have very low overhead for common code (almost no contention), almost nothing changed for uniq directory operations. At the same time, we decreased contention on pdirops because changes on shared directory are localized on a few CPUs.</p> <p>Also, we probably can sort unlink requests inside MDD schedulers to decrease disk seek somehow, but I'm not sure how much it can help </p> <p>I need to make some changes to libcfs_cpu.c because we can't always support numa correctly because current implementation is based on assumption that all cores of a numa-node have contiguous core-id, which is not true:</p> <p>numactl -H<br/> available: 4 nodes (0-3)<br/> node 0 cpus: 0 4 8 12 16 20 24 28<br/> node 0 size: 16364 MB<br/> node 0 free: 13507 MB<br/> node 1 cpus: 1 5 9 13 17 21 25 29<br/> node 1 size: 16384 MB<br/> node 1 free: 15534 MB<br/> node 2 cpus: 2 6 10 14 18 22 26 30<br/> node 2 size: 16384 MB<br/> node 2 free: 15558 MB<br/> node 3 cpus: 3 7 11 15 19 23 27 31<br/> node 3 size: 16382 MB<br/> node 3 free: 15911 MB<br/> node distances:<br/> node 0 1 2 3<br/> 0: 10 21 21 21<br/> 1: 21 10 21 21<br/> 2: 21 21 10 21<br/> 3: 21 21 21 10</p> <p>this machine has 4 numa nodes and 32 cores, i.e: if we want to configure it as 16 libcfs cpu-nodes, then we will bind (0,1), (2,3), .... (30,31), but I would expect the pattern like: (0, 4), (8, 12), ....(27, 31)</p> <p>Hi,</p> <p>Indeed, on our MESCA nodes this is the same code-id distribution. So, I am afraid we will systematically get wrong cpu binding if we test liang/b_smp.</p> <p>I gave a look at the code, it seems to me that the cpu-nodes binding is done in function cfs_cpumap_setup_numa(). But unfortunately I cannot see how to change current implementation in order to take into account our code-id distribution.<br/> Could you give some guidance on what you think should be the best way to address this?</p> <p>TIA,<br/> Sebastien.</p> <p>Sebastien,</p> <p>if you want to use the branch on OSS or client, please use these modparams:<br/> libcfs cpu_mode=0 // "0" is for NUMA <br/> lnet network="o2ib0(ib0:0),o2ib(ib1:1)..." // number after colon is NUMA ID, probably I should move it after the bracket in the future..., by this way, the interface is bound on the NUMA node and all traffic should be localized on the NUMA node.</p> <p>Regards<br/> Liang</p> <p>Hi Liang,</p> <p>Thanks for the information.<br/> By NUMA ID, do you mean the same number as the node number given by 'numactl -H' command?</p> <p>TIA,<br/> Sebastien.</p> <p>it's "logic" ID, I don't know if it's possible to have non-contiguous physical numa ID, but even it's possible we still use 0, 1, 2, ... as "logic" ID, which has same order as physical numa ID. </p> <p>I've made a little change to the way of binding CPU on NI, now it's like<br/> lnet networks="o2ib0:0(ib0), o2ib1:1(ib1)..."</p> <p>Regards<br/> Liang</p> <p>Just add a comment to introduce the way of tuning b_smp:</p> <ul> <li>concept: cpu_node can present any number of physical cores (or hyper-threading), or a NUMA node</li> <li>lustre will have thread pool and message queue for each cpu_node, instead of current all globals, to get better concurrency and reduce migration data/thread between CPUs.</li> <li>there are three "cpu_mode" for libcfs <ul> <li>cpu_mode=0: (NUMA) I've changed it to be default mode, by this mode, a libcfs cpu_node is presented by one physical NUMA node, OSS and client should use this mode</li> <li>cpu_mode=1: (UMA) it's just for debug, with this mode, user can specify number of cpu nodes by "libcfs cpu_node_num=NUMBER", NUMBER can be (1 - N), N is total number of physical cores, or total number of hyper-threadings if hyper-threading is enabled. This mode is really reserved for debug or testing</li> <li>cpu_mode=2: (FAT) this mode is for MDS, it will estimate proper number of cpu nodes automatically, which is based on total cores and RAM size, however, the estimating is not perfect because modern CPUs can have different number of cores, i.e: 2-cores, 4-cores, 6-cores, 8-cores..., so before I have a better solution, I would suggest to configure it manually (sorry I know it's stupid...), here is suggested configuration cpu_node number on MDS: <ul> <li>4-6 cores: cpu_node_num=2</li> <li>8-24 cores: cpu_node_num=4</li> <li>more cores: cpu_node_num=8</li> </ul> </li> </ul> </li> </ul> <p>Testing results show pdirops patch works very good (<a href="https://jira.whamcloud.com/browse/LU-50" title="pdirops patch for ldiskfs" class="issue-link" data-issue-key="LU-50"><del>LU-50</del></a>), so I will remove "MDD dirops cache &amp; transaction schedulers" from my branch very soon.</p> <p>Hi Liang,</p> <p>I am currently trying to test your liang/b_smp branch on 2.6.32, on one of our MESCA servers.<br/> The problem is my server crashes when I try to 'modprobe ptlrpc', with the following output on the console:</p> <p><span class="error">&#91;root@berlin7 ~&#93;</span># modprobe ptlrpc<br/> BUG: unable to handle kernel paging request at 000000000000ea6c<br/> IP: <span class="error">&#91;&lt;ffffffffa04418df&gt;&#93;</span> cfs_cpu_hwnode+0x2f/0x70 <span class="error">&#91;libcfs&#93;</span><br/> PGD c6d23a067 PUD c5202e067 PMD 0<br/> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu31/cache/index2/shared_cpu_map<br/> CPU 6<br/> Modules linked in: libcfs<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/>(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache nfs_acl auth_rpcgss sunrpc rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) mlx4_core(U) dm_round_robin dm_multipath sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma lpfc scsi_transport_fc scsi_tgt igb dca ext3 jbd sd_mod crc_t10dif usbhid hid ahci ehci_hcd uhci_hcd dm_mod <span class="error">&#91;last unloaded: microcode&#93;</span></p> <p>Modules linked in: libcfs<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/>(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache nfs_acl auth_rpcgss sunrpc rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) mlx4_core(U) dm_round_robin dm_multipath sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma lpfc scsi_transport_fc scsi_tgt igb dca ext3 jbd sd_mod crc_t10dif usbhid hid ahci ehci_hcd uhci_hcd dm_mod <span class="error">&#91;last unloaded: microcode&#93;</span><br/> Pid: 38400, comm: modprobe Not tainted 2.6.32-71.14.1.el6.Bull.18.x86_64 #1 bullx super-node<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa04418df&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa04418df&gt;&#93;</span> cfs_cpu_hwnode+0x2f/0x70 <span class="error">&#91;libcfs&#93;</span><br/> RSP: 0018:ffff880434711d18 EFLAGS: 00010246<br/> RAX: 0000000000000000 RBX: 0000000000000080 RCX: 0000000000000014<br/> RDX: 000000000000ea6c RSI: 00000000ffffffff RDI: 0000000000000000<br/> RBP: ffff880434711d18 R08: ffff880434711e98 R09: 0000000000000000<br/> R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000000<br/> R13: 0000000000000014 R14: 0000000000000050 R15: 00000000ffffffff<br/> FS: 00007f6db66a0700(0000) GS:ffff88088e440000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br/> CR2: 000000000000ea6c CR3: 0000000c6cf16000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process modprobe (pid: 38400, threadinfo ffff880434710000, task ffff88046cee4f60)<br/> Stack:<br/> ffff880434711d58 ffffffffa0441ab2 ffff880434711d98 ffffffff8171dfe0<br/> &lt;0&gt; 0000000000000004 0000000000000004 0000000000000004 0000000000000000<br/> &lt;0&gt; ffff880434711d68 ffffffffa0441b10 ffff880434711dd8 ffffffffa0455ad0<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0441ab2&gt;&#93;</span> __cfs_node_alloc+0x42/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0441b10&gt;&#93;</span> __cfs_node_alloc_aligned+0x20/0x30 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0455ad0&gt;&#93;</span> cfs_cpumap_alloc+0x50/0x590 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8125c240&gt;&#93;</span> ? __bitmap_weight+0x50/0xb0<br/> <span class="error">&#91;&lt;ffffffffa04563ec&gt;&#93;</span> cfs_cpumap_create_numa+0x12c/0x2e0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0448f20&gt;&#93;</span> ? init_libcfs_module+0x0/0x270 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0456623&gt;&#93;</span> cfs_cpu_init+0x83/0x290 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa044cd89&gt;&#93;</span> ? cfs_tracefile_init+0xd9/0x130 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81084a85&gt;&#93;</span> ? atomic_notifier_chain_register+0x55/0x60<br/> <span class="error">&#91;&lt;ffffffffa044074c&gt;&#93;</span> ? libcfs_register_panic_notifier+0x1c/0x20 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0445e90&gt;&#93;</span> ? libcfs_debug_init+0x90/0x100 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0448f20&gt;&#93;</span> ? init_libcfs_module+0x0/0x270 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0448fc5&gt;&#93;</span> init_libcfs_module+0xa5/0x270 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81084805&gt;&#93;</span> ? __blocking_notifier_call_chain+0x65/0x80<br/> <span class="error">&#91;&lt;ffffffff8100904c&gt;&#93;</span> do_one_initcall+0x3c/0x1d0<br/> <span class="error">&#91;&lt;ffffffff8109d3af&gt;&#93;</span> sys_init_module+0xdf/0x250<br/> <span class="error">&#91;&lt;ffffffff8100c172&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> Code: 0f 1f 44 00 00 48 85 ff 89 f0 74 0e 85 f6 78 05 3b 77 40 72 37 b8 ff ff ff ff 48 98 48 c7 c2 6c ea 00 00 48 8b 04 c5 40 d4 71 81 &lt;8b&gt; 04 10 0f a3 05 7f e9 2d e1 19 d2 85 d2 ba ff ff ff ff 0f 44<br/> RIP <span class="error">&#91;&lt;ffffffffa04418df&gt;&#93;</span> cfs_cpu_hwnode+0x2f/0x70 <span class="error">&#91;libcfs&#93;</span><br/> RSP &lt;ffff880434711d18&gt;</p> <p>For your information, I have not set manually the 'cpu_mode' option of libcfs module (so it should be 0).</p> <p>Have you already seen this?<br/> Any idea to fix it?</p> <p>TIA,<br/> Sebastien.</p> <p>Sebastien, sorry I've never seen this and actually I never tried my branch on 2.6.32, but I will look into this issue very soon, btw, are you able to apply the new kernel patch to 2.6.32? (ext4_pdirop.patch)</p> <p>Yes, after small adjustments in patch, I see that ext4_pdirop.patch is applied at compile time in order to build ldiskfs.</p> <p>I hoped we could run some obdfilter-survey tests at least, in order to have nice figures to present at the LUG <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>For information, the problem occurs on the first instruction of cfs_cpumap_alloc() function, i.e.<br/> LIBCFS_ALLOC_ALIGNED(cpumap, sizeof(cfs_cpumap_t));</p> <p>Hi, it should be fixed, it's a typo which will make my code access cpu_to_node<span class="error">&#91;-1&#93;</span>...it's been there for months but I never hit it on many different hardwares... please pull from git for the fix. <br/> btw: I've changed default value of cpu_mode to 2 for my testing, so please use cpu_mode=0 to modprobe.conf on OSS or clients. </p> <p>Also, binding LNet on CPU are changed as well... now it should be like "o2ib0:0(ib0), o2ib1:1(ib1)"</p> <p>Hi Liang,</p> <p>Thanks for the quick fix, now I can 'modprobe ptlrpc' without any issue.</p> <p>But unfortunately I am hitting another problem: when I try to mount the MDT, I get the following error:</p> <p><span class="error">&#91;root@berlin27 lustre&#93;</span># mount -t lustre -o acl,user_xattr /dev/sda9 /mnt/t100full/mdt/0<br/> mount.lustre: mount /dev/sda9 at /mnt/t100full/mdt/0 failed: Invalid argument<br/> This may have multiple causes.<br/> Are the mount options correct?<br/> Check the syslog for more info.</p> <p><span class="error">&#91;root@berlin27 lustre&#93;</span># dmesg -c<br/> LDISKFS-fs warning (device sda9): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> LDISKFS-fs (sda9): mounted filesystem with ordered data mode<br/> LDISKFS-fs warning (device sda9): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> LDISKFS-fs (sda9): Unrecognized mount option "64bithash" or missing value<br/> LustreError: 16785:0:(obd_mount.c:1399:server_kernel_mount()) ll_kern_mount failed: rc = -22<br/> LustreError: 16785:0:(obd_mount.c:1680:server_fill_super()) Unable to mount device /dev/sda9: -22<br/> LustreError: 16785:0:(obd_mount.c:2154:lustre_fill_super()) Unable to mount (-22)</p> <p>Did I miss something? Like a specific e2fsprogs version (we are currently using 1.41.10.sun2)?</p> <p>I must say that we do not have this error if we start the same MDT with our Lustre 2.0.0.1 rpms installed.</p> <p>TIA,<br/> Sebastien.</p> <p>Hi, I think you need this patch: <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=6bdb62e3d7ed206bcaef0cd3499f9cb56dc1fb92" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=6bdb62e3d7ed206bcaef0cd3499f9cb56dc1fb92</a><br/> I've synced my branch with master, so you just need to pull it from my branch. </p> <ul> <li>Background of the project <ul> <li>already landed tens of standalone patches when I was in Oracle, all following descriptions are about patches not landed yet</li> <li>A fat server is divided into several processing partitions (or cpu-partitions, CP as abbreviation), each partition contains: some cpu cores(or NUMA nodes) + memory pool + message queue + threads pool), it's a little like concept of virtual machine, although it's much simpler. It's not new thing, I just bring in this as a concept to replace what I used to call cpu_node + cpu-affinity, which is confusing because "node" is already used to present NUMA in linux kernel.</li> <li>Although we still have one single namespace on MDS, but we can have several virtual processing partitions, lifetime of request should be localized on the processing partition as possible as we can, to reduce data/thread migration between CPUs, also reduce lock contentions and cacheline conflicts. <ul> <li>LND has message queue and threads-pool for each partition</li> <li>LNet has EQ callback for each partition</li> <li>Ptlrpc service has message queue and threads pool for each partition</li> </ul> </li> <li>number of CPU partitions is configurable (by new parameter "cpu_npartitions" of libcfs), libcfs will also automatically estimate "cpu_npartitions" based on number of CPUs. If we have cpu_node_num=N (N &gt; 1), Lustre stack can have N standalone processing flows (unless they are contenting on the same resource), if we configured the system with cpu_npartitions=1, we have only one partition and lustre should act as current "master".</li> <li>user can provide string pattern for cpu-partitions as well, i.e: "0[0, 2, 4, 6] 1[1, 3, 5, 7]", by this way, lustre will have two cpu-partitions, the first one contains core[0, 2, 4, 6], the second one presents core[1, 3, 5, 7], NB: those numbers inside bracket can be NUMA id as well.</li> <li>number of cpu partitions &lt; number of cores (or hyper-threadings), because modern computer can have hundreds or more cores, there are some major downsides if we have per-core schedulers(threads) pool: a) bad load balance especially on small/middle size cluster, b) too many threads overall; <ul> <li>on client/OSS/router, cpu_npartitions == NUMA node, dataflow of request/reply is localized on NUMA node</li> <li>on MDS, a cpu partition will present a few cores(or hyper-threading), i.e: on system with 4 NUMA nodes and 64-cores, we have 8 cpu-partitions</li> <li>we might hash different objects(by fid) to different cpu-partition in the future, it's an advanced feature which we don't have now</li> <li>we can bind LNet NI on specified cpu partition for NUMIOA performance</li> </ul> </li> <li>these things helped a lot on performance of many target directories tests, but they can help nothing on shared directory performance at all, I never saw 20K+ file/sec in shared directory creatinon/removal tests (non-zero stripe), with 1.8, the number is &lt; 7K files/sec</li> <li>new pdirops patch works fine(<a href="https://jira.whamcloud.com/browse/LU-50" title="pdirops patch for ldiskfs" class="issue-link" data-issue-key="LU-50"><del>LU-50</del></a>), Andreas has already reviewed the prototype, now I have posted an the second version for review. The pdirops patch is different with the old one (we had 5 or 6 years ago?) <ul> <li>the old version is dynlock based, it has too many inline changes for ext4, and we probably need to change more to support N-level htree (large directory), we probably have to change ext4/namei.c and make it quite like our IAM.</li> <li>the new version is htree_lock based, although implementation of the htree_lock is big &amp; complex, but it just requires a few inline changes for ext4. htree_lock is more like a embedded component, it can be enabled/disabled easily, and need very few logic change to current ext4. The patch is big (about 2K lines), but only has about 200 LOC inline changes (including changes for N-level htree), and half of those inline changes are adding a new parameter to some functions.</li> <li>htree_lock based pdirops patch have same performance as IAM dir, but without any interop issue</li> <li>with pdirops patch on my branch, we can get 65K+ files/sec opencreate+close (narrow stripping file, aggregation performance on server) in my latest test on toro</li> </ul> </li> <li>MDD transaction scheduler will be totally dropped, If you still remember it, I mentioned previously that I wanted to add a relatively small threads pool in MDD to take over backend filesystem transactions, this thread pool size should be moderate and just enough for driving full throughput of metadata operations, this idea is majorly for share directory performance. However, pdirops patch works very good, so I will totally drop it (there is more detail in my mail to lustre-devel a few weeks ago)</li> </ul> </li> <li>key component for landing <ul> <li>Tens of standalone patches <ul> <li>no dependency on each other, size of them are from a few LOC to a few hundreds LOC</li> <li>some of them will be landed on 2.1, all of them will be landed on 2.2</li> </ul> </li> <li>pdirop patch + BH LRU size kernel patch <ul> <li>pdirop patch is big, but easy to maintain (at least it's easy to port it to rhel6)</li> <li>I've sent another mail to Andreas and bzzz to explain why we need to increase size of BH LRU size, it will be a small patch</li> <li>we would like to land them on 2.2 if we got enough resource or got funding on this</li> </ul> </li> <li>cpu-partition patches <ul> <li>it's the biggest chunk, includes several large patches spread over stack layers (libcfs, LNet, LND, ptlrpc server side, and some small changes to other modules like mdt, ost )</li> <li>patches have dependencies.</li> <li>The biggest challenge is inspection LNet + LNDs, isaac might give us some help to review, Lai Siyao will be the other inspector</li> <li>if we got funding on this, we tend to land them on 2.2, otherwise it's more realistic for us to land them on 2.3</li> </ul> </li> </ul> </li> </ul> <p>Can you post links to your patches here. Thank you.</p> <p><a href="http://review.whamcloud.com/#change,2346" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2346</a><br/> <a href="http://review.whamcloud.com/#change,2461" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2461</a><br/> <a href="http://review.whamcloud.com/#change,2523" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2523</a><br/> <a href="http://review.whamcloud.com/#change,2558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2558</a> <br/> <a href="http://review.whamcloud.com/#change,2638" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2638</a><br/> <a href="http://review.whamcloud.com/#change,2718" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2718</a><br/> <a href="http://review.whamcloud.com/#change,2725" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2725</a><br/> <a href="http://review.whamcloud.com/#change,2729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2729</a></p> <p>FYI - They're still in review.</p> <p>Re: the last two comments. Patches have landed. I'll copy an update of the patch list from Richard Henwood here:</p> <p>c3e985e7e98f41ebf5ecb78887dcd2554601f7ef <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: post rqbd with flag LNET_INS_LOCAL<br/> 8bbd62a7c0d2fc48d8f11e78d92bb42809968bba <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: CPT affinity ptlrpc RS handlers<br/> d800fc41a1abdaf7aaf6c0e3e7ddcdec489985a8 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: partitioned ptlrpc service<br/> b43a6b1800265608cfa18159d4d0d006a1c23015 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> o2iblnd: CPT affinity o2iblnd<br/> 82e02a17c0c645a8d156e51b8d8da5eaa68b8f5b <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: re-finalize failed ACK or routed message<br/> 1a73553d15b459208cbf7279ea6e5e5a110c632b <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ksocklnd: CPT affinity socklnd<br/> 6b4b780895dfdeaca316862fbf1696983608f96d <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: wrong assertion for optimized GET<br/> 913c8e22cfc4fc5f52c4f0d6d3f0b4b86a7ac58c <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: tuning wildcard portals rotor<br/> c03783fce46ae0b40db0680388df6e2d6fca5008 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: SMP improvements for LNet selftest<br/> 7b2ab9beae02080797ff2da5105eaddadd67c151 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ldlm: SMP improvement for ldlm_lock_cancel<br/> 07b8db220e48782369f48d86213c5d404a628ded <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: Reduce at_lock dance<br/> c48a869557fe7663f4f3370b130d4c248958180e <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: CPT affinity workitem scheduler<br/> 8a5b8dbda960b155f669c13602504f1233a84c7e <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> obdclass: SMP improvement for lu_key<br/> e531dc437c56a08a65de9074a511faa55184712b <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: multiple cleanups for inspection<br/> e069296630240947f1815e505067fd48033909f7 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: allow user to bind NI on CPTs<br/> a07e9d350b3e500c7be877f6dcf54380b86a9cbe <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: Partitioned LNet networks<br/> 5e1957841df3e771f3d72d8ea59180213430bbb9 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: cleanup for rtrpool and LNet counter<br/> 279bbc81e03dc74d273ec12b4d9e703ca94404c4 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: Partitioned LNet resources (ME/MD/EQ)<br/> 582c110231cf06bcd7e5e0b3bdf4f2058e18ebe4 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: cleanup of ptlrpc_unregister_service<br/> ff0c89a73e141ce019ee2a94e5d01a8a37dd830a <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: svc thread starting/stopping cleanup<br/> 25766da50b627648b04549ff3fb55af12acbcb4b <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: reduce stack usage of "match" functions<br/> c7bff5640caff778d4cfca229672a2cc67b350d6 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: Granulate LNet lock<br/> 24564b398f53009521aeda5d653e57fe8b525775 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: partition data for ptlrpc service<br/> 698d3088622b4610a84bd508f2b707a7a2dd1e3e <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: code cleanup for lib-move.c<br/> 38fcdd3966da09517ca176b962230b7dae43514c <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: match-table for Portals<br/> f0aa1eef72e7438c2bd4b3eee821fefbc50d1f8e <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: code cleanup for lib-md.c<br/> 75a8f4b4aa9ad6bf697aedece539e62111e9029a <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: split lnet_commit_md and cleanup<br/> 06093c1f24da938418a0243259b5307c9fc338d5 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: LNet message event cleanup<br/> 2118a8b92cec2df85d1bdbe2e58b389d83fe06b2 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: eliminate a few locking dance in LNet<br/> 51a5b4df5bbbf5fd12c73d2722b230e93fe93327 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: parse RC ping in event callback<br/> b9bad9bd7d1c3271df916ee62091106e3f3c98b7 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: router-checker (RC) cleanup<br/> 4fcc56be68c8c1667fbd91721d084874a2f05c3e <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: common code to validate nthreads<br/> ed22093b2d569fd0e93f35504580171114bf212d <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: move "match" functions to lib-ptl.c<br/> a096d858b671f28fd4c5e6197b51643cd0780a50 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: allow to create EQ with zero eq_size<br/> c1366da8f43ecfb98ef3bdcf629eec8a2fc9cd4c <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: cleanup for LNet Event Queue<br/> 3211f6862cbbe96642db540e6593f3c614f9528c <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: new internal object lnet_peer_table<br/> 7a51ad347960ef2b9d1dfad14644c0bca35b80b6 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> ptlrpc: clean up ptlrpc svc initializing APIs<br/> facf5086667874c405c9ef6ce7f8f737868ffefd <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: container for LNet message<br/> c3a57ec36441c75df03cfbec8f718e053aaad12a <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: abstract container for EQ/ME/MD<br/> 4bd9bf53728260d38efc74cac981318fe31280cd <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> lnet: add lnet_*_free_locked for LNet<br/> c8da7bfbe0505175869973b25281b152940774b0 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: more common APIs in libcfs<br/> b76f327de7836a854f204d28e61de52bc03011b1 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: export a few symbols from libcfs<br/> 3a92c850b094019e556577ec6cab5907538dcbf5 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: NUMA allocator and code cleanup<br/> 617e8e1229637908d4cce6725878dd5668960420 <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: implementation of cpu partition<br/> 19ec037c0a9427250b87a69c53beb153d533ab1c <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> libcfs: move range expression parser to libcfs</p> <p>Cray has been testing (a little) for functionality only. No significant issues to report so far.</p> <p>SMP node affinity presentation for Lustre team Tech-call</p> <p>SMP node affinity OpenSFS demonstration</p> <p>Please let me know if more work is needed for this and I will reopen ticket.</p> <p>patched mdtest which parses expression for multi-mount</p> Related LU-14676 LUDOC-97 Development Rank 1|hzv6cv: Rank (Obsolete) 4564