I've merged NUMA code into SMP patches of LNet and ptlrpc service in last week, will start to post patches soon.

I'm pushing changes into liang/b_hd_smp, those patches can be landed in weeks will not be pushed in.
Also, I'm working on another three issues:

1. move lprocfs_counter to libcfs
   • current lprocfs_counter is not very efficient and has a small defect
   • we want to use percpu counter inside libcfs and LNet
2. general purpose threads pool APIs
   • several modules of Lustre have their own implementation to create/destroy thread pool, if we have a general purpose thread pool implementation, we can get ride of some code redundance.
3. MDD threads pool for create/link/unlink
   • As we know, on large scale SMP system, SMP scalability of diskfs will be bottleneck of these operations under shared directory, there is no easy solution of this.
   • I had a discussion with Oleg last week, we got agreement about design MDD threads pool like this way:
     • MDD has a threads pool, all threads are CPU affinity
     • MDD also has two levels caches
       • percpu fid cache
         parent fid of create/unlink/link will be added to top local CPU cache, if it's already in this cache, it just pop it up and increase refcount on the parent_fid, the fid can be discarded because cache has limited size (i.e: 8)
       • global fid cache
         if refcount on the fid is larger than a value (i.e: 128), we also enqueue the fid into a global cache, if we found the fid is already in the global cache, we just increase refcount on the global cached fid. When refcount on global cached fid is larger than a specified value N(i.e: N=4), which means, there are "N" cores are create/link/unlink a lot of files under the same directory, we allocate 1 or 2 cores for this parent fid, all FS operations like create/link/unlink will be dispatched to MDD thread on these 1 or 2 cores.
     • if the parent fid is not in global cache, or global refcount is lower than specified value, we still process FS operations on thread context of caller
     • we can easily enable/disable the caches for one or all operations.
     • cache size is also configurable

Liang

Liang, I don't quite understand what you mean about "patches can be landed in weeks will not be pushed in." We need a local branch with all the SMP patches in it so we can start testing this at LLNL. Can we use liang/b_hd_smp for this?

Robert, Yujian & Vitaly are in progress of testing & landing some patches, I think they can be done in weeks, so I think probably we don't have to push them into b_hd_smp because they will be in main branch very soon.

All patches are already in branch liang/b_hd_smp, prototype of mdd thread pool + dir cache are already in the branch.
Jim is going to test with it.

made some improvements to mdd dir cache + transaction threads pool, I will post the whole idea at here later.

I've created another ticket to track improvements on la_get (LU-36)

Description of MDD dirops cache & transaction schedulers

• each CPU has a local fid cache (it's a 8 buckets hash-table, each bucket is a 8 items LRU, so we can cache 64 fids by default, of course we can increase number of buckets)
• transaction schedulers are percpu threads pool in MDD layer
• each time we create/remove file from a directory, we push the fid into the local CPU fid cache(LRU)
• if the fid is already in the cache, then just increase a refcount(# of hits) on the fid and move the fid to top of LRU
• when we push a new fid into local cache, the fid at bottom of LRU will be popped out
• if refcount(# of hits) of fid in local CPU cache >= N (default is 32), we try to lookup the fid in global fid cache, which is a hash-table as well (do nothing if it already has reference to fid in global cache).
  • if the fid is not in the global cache, we just add it to global cache and set refcount to 1 (1 CPU is using it)
  • if the fid is already in global cache, we increase refcount of fid in global cache
• if refcount of fid in global cache < M, we just return -1 and continue to run transaction in context of current thread (mdt service thread)
• if refcount of fid in global cache is >= M (M is number of CPUs, which can be 2, 4... as many as we can get good performance of shared directory with ldiskfs/ext4, it's tunable), we specify M CPUs for the fid and return one CPU id within those M CPUs, if caller get a CPU id different with current CPU id, it will launch a MDD level "transaction request", and wakeup one MDD transaction thread to handle the "transaction request"

By this way, we only have very low overhead for common code (almost no contention), almost nothing changed for uniq directory operations. At the same time, we decreased contention on pdirops because changes on shared directory are localized on a few CPUs.

Also, we probably can sort unlink requests inside MDD schedulers to decrease disk seek somehow, but I'm not sure how much it can help

I need to make some changes to libcfs_cpu.c because we can't always support numa correctly because current implementation is based on assumption that all cores of a numa-node have contiguous core-id, which is not true:

numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 4 8 12 16 20 24 28
node 0 size: 16364 MB
node 0 free: 13507 MB
node 1 cpus: 1 5 9 13 17 21 25 29
node 1 size: 16384 MB
node 1 free: 15534 MB
node 2 cpus: 2 6 10 14 18 22 26 30
node 2 size: 16384 MB
node 2 free: 15558 MB
node 3 cpus: 3 7 11 15 19 23 27 31
node 3 size: 16382 MB
node 3 free: 15911 MB
node distances:
node 0 1 2 3
0: 10 21 21 21
1: 21 10 21 21
2: 21 21 10 21
3: 21 21 21 10

this machine has 4 numa nodes and 32 cores, i.e: if we want to configure it as 16 libcfs cpu-nodes, then we will bind (0,1), (2,3), .... (30,31), but I would expect the pattern like: (0, 4), (8, 12), ....(27, 31)

Hi,

Indeed, on our MESCA nodes this is the same code-id distribution. So, I am afraid we will systematically get wrong cpu binding if we test liang/b_smp.

I gave a look at the code, it seems to me that the cpu-nodes binding is done in function cfs_cpumap_setup_numa(). But unfortunately I cannot see how to change current implementation in order to take into account our code-id distribution.
Could you give some guidance on what you think should be the best way to address this?

TIA,
Sebastien.

Sebastien,

if you want to use the branch on OSS or client, please use these modparams:
libcfs cpu_mode=0 // "0" is for NUMA
lnet network="o2ib0(ib0:0),o2ib(ib1:1)..." // number after colon is NUMA ID, probably I should move it after the bracket in the future..., by this way, the interface is bound on the NUMA node and all traffic should be localized on the NUMA node.

Regards
Liang

Hi Liang,

Thanks for the information.
By NUMA ID, do you mean the same number as the node number given by 'numactl -H' command?

TIA,
Sebastien.

it's "logic" ID, I don't know if it's possible to have non-contiguous physical numa ID, but even it's possible we still use 0, 1, 2, ... as "logic" ID, which has same order as physical numa ID.

I've made a little change to the way of binding CPU on NI, now it's like
lnet networks="o2ib0:0(ib0), o2ib1:1(ib1)..."

Regards
Liang

Just add a comment to introduce the way of tuning b_smp:

• concept: cpu_node can present any number of physical cores (or hyper-threading), or a NUMA node
• lustre will have thread pool and message queue for each cpu_node, instead of current all globals, to get better concurrency and reduce migration data/thread between CPUs.
• there are three "cpu_mode" for libcfs
  • cpu_mode=0: (NUMA) I've changed it to be default mode, by this mode, a libcfs cpu_node is presented by one physical NUMA node, OSS and client should use this mode
  • cpu_mode=1: (UMA) it's just for debug, with this mode, user can specify number of cpu nodes by "libcfs cpu_node_num=NUMBER", NUMBER can be (1 - N), N is total number of physical cores, or total number of hyper-threadings if hyper-threading is enabled. This mode is really reserved for debug or testing
  • cpu_mode=2: (FAT) this mode is for MDS, it will estimate proper number of cpu nodes automatically, which is based on total cores and RAM size, however, the estimating is not perfect because modern CPUs can have different number of cores, i.e: 2-cores, 4-cores, 6-cores, 8-cores..., so before I have a better solution, I would suggest to configure it manually (sorry I know it's stupid...), here is suggested configuration cpu_node number on MDS:
    • 4-6 cores: cpu_node_num=2
    • 8-24 cores: cpu_node_num=4
    • more cores: cpu_node_num=8

Testing results show pdirops patch works very good (LU-50), so I will remove "MDD dirops cache & transaction schedulers" from my branch very soon.

Hi Liang,

I am currently trying to test your liang/b_smp branch on 2.6.32, on one of our MESCA servers.
The problem is my server crashes when I try to 'modprobe ptlrpc', with the following output on the console:

[root@berlin7 ~]# modprobe ptlrpc
BUG: unable to handle kernel paging request at 000000000000ea6c
IP: [<ffffffffa04418df>] cfs_cpu_hwnode+0x2f/0x70 [libcfs]
PGD c6d23a067 PUD c5202e067 PMD 0
Oops: 0000 1 SMP
last sysfs file: /sys/devices/system/cpu/cpu31/cache/index2/shared_cpu_map
CPU 6
Modules linked in: libcfs(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache nfs_acl auth_rpcgss sunrpc rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) mlx4_core(U) dm_round_robin dm_multipath sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma lpfc scsi_transport_fc scsi_tgt igb dca ext3 jbd sd_mod crc_t10dif usbhid hid ahci ehci_hcd uhci_hcd dm_mod [last unloaded: microcode]

Modules linked in: libcfs(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache nfs_acl auth_rpcgss sunrpc rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) mlx4_en(U) mlx4_core(U) dm_round_robin dm_multipath sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma lpfc scsi_transport_fc scsi_tgt igb dca ext3 jbd sd_mod crc_t10dif usbhid hid ahci ehci_hcd uhci_hcd dm_mod [last unloaded: microcode]
Pid: 38400, comm: modprobe Not tainted 2.6.32-71.14.1.el6.Bull.18.x86_64 #1 bullx super-node
RIP: 0010:[<ffffffffa04418df>] [<ffffffffa04418df>] cfs_cpu_hwnode+0x2f/0x70 [libcfs]
RSP: 0018:ffff880434711d18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000080 RCX: 0000000000000014
RDX: 000000000000ea6c RSI: 00000000ffffffff RDI: 0000000000000000
RBP: ffff880434711d18 R08: ffff880434711e98 R09: 0000000000000000
R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000014 R14: 0000000000000050 R15: 00000000ffffffff
FS: 00007f6db66a0700(0000) GS:ffff88088e440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000ea6c CR3: 0000000c6cf16000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 38400, threadinfo ffff880434710000, task ffff88046cee4f60)
Stack:
ffff880434711d58 ffffffffa0441ab2 ffff880434711d98 ffffffff8171dfe0
<0> 0000000000000004 0000000000000004 0000000000000004 0000000000000000
<0> ffff880434711d68 ffffffffa0441b10 ffff880434711dd8 ffffffffa0455ad0
Call Trace:
[<ffffffffa0441ab2>] __cfs_node_alloc+0x42/0x80 [libcfs]
[<ffffffffa0441b10>] __cfs_node_alloc_aligned+0x20/0x30 [libcfs]
[<ffffffffa0455ad0>] cfs_cpumap_alloc+0x50/0x590 [libcfs]
[<ffffffff8125c240>] ? __bitmap_weight+0x50/0xb0
[<ffffffffa04563ec>] cfs_cpumap_create_numa+0x12c/0x2e0 [libcfs]
[<ffffffffa0448f20>] ? init_libcfs_module+0x0/0x270 [libcfs]
[<ffffffffa0456623>] cfs_cpu_init+0x83/0x290 [libcfs]
[<ffffffffa044cd89>] ? cfs_tracefile_init+0xd9/0x130 [libcfs]
[<ffffffff81084a85>] ? atomic_notifier_chain_register+0x55/0x60
[<ffffffffa044074c>] ? libcfs_register_panic_notifier+0x1c/0x20 [libcfs]
[<ffffffffa0445e90>] ? libcfs_debug_init+0x90/0x100 [libcfs]
[<ffffffffa0448f20>] ? init_libcfs_module+0x0/0x270 [libcfs]
[<ffffffffa0448fc5>] init_libcfs_module+0xa5/0x270 [libcfs]
[<ffffffff81084805>] ? __blocking_notifier_call_chain+0x65/0x80
[<ffffffff8100904c>] do_one_initcall+0x3c/0x1d0
[<ffffffff8109d3af>] sys_init_module+0xdf/0x250
[<ffffffff8100c172>] system_call_fastpath+0x16/0x1b
Code: 0f 1f 44 00 00 48 85 ff 89 f0 74 0e 85 f6 78 05 3b 77 40 72 37 b8 ff ff ff ff 48 98 48 c7 c2 6c ea 00 00 48 8b 04 c5 40 d4 71 81 <8b> 04 10 0f a3 05 7f e9 2d e1 19 d2 85 d2 ba ff ff ff ff 0f 44
RIP [<ffffffffa04418df>] cfs_cpu_hwnode+0x2f/0x70 [libcfs]
RSP <ffff880434711d18>

For your information, I have not set manually the 'cpu_mode' option of libcfs module (so it should be 0).

Have you already seen this?
Any idea to fix it?

TIA,
Sebastien.

Sebastien, sorry I've never seen this and actually I never tried my branch on 2.6.32, but I will look into this issue very soon, btw, are you able to apply the new kernel patch to 2.6.32? (ext4_pdirop.patch)

Yes, after small adjustments in patch, I see that ext4_pdirop.patch is applied at compile time in order to build ldiskfs.

I hoped we could run some obdfilter-survey tests at least, in order to have nice figures to present at the LUG

For information, the problem occurs on the first instruction of cfs_cpumap_alloc() function, i.e.
LIBCFS_ALLOC_ALIGNED(cpumap, sizeof(cfs_cpumap_t));

Hi, it should be fixed, it's a typo which will make my code access cpu_to_node[-1]...it's been there for months but I never hit it on many different hardwares... please pull from git for the fix.
btw: I've changed default value of cpu_mode to 2 for my testing, so please use cpu_mode=0 to modprobe.conf on OSS or clients.

Also, binding LNet on CPU are changed as well... now it should be like "o2ib0:0(ib0), o2ib1:1(ib1)"

Hi Liang,

Thanks for the quick fix, now I can 'modprobe ptlrpc' without any issue.

But unfortunately I am hitting another problem: when I try to mount the MDT, I get the following error:

[root@berlin27 lustre]# mount -t lustre -o acl,user_xattr /dev/sda9 /mnt/t100full/mdt/0
mount.lustre: mount /dev/sda9 at /mnt/t100full/mdt/0 failed: Invalid argument
This may have multiple causes.
Are the mount options correct?
Check the syslog for more info.

[root@berlin27 lustre]# dmesg -c
LDISKFS-fs warning (device sda9): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sda9): mounted filesystem with ordered data mode
LDISKFS-fs warning (device sda9): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sda9): Unrecognized mount option "64bithash" or missing value
LustreError: 16785:0:(obd_mount.c:1399:server_kernel_mount()) ll_kern_mount failed: rc = -22
LustreError: 16785:0:(obd_mount.c:1680:server_fill_super()) Unable to mount device /dev/sda9: -22
LustreError: 16785:0:(obd_mount.c:2154:lustre_fill_super()) Unable to mount (-22)

Did I miss something? Like a specific e2fsprogs version (we are currently using 1.41.10.sun2)?

I must say that we do not have this error if we start the same MDT with our Lustre 2.0.0.1 rpms installed.

TIA,
Sebastien.

Hi, I think you need this patch: http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=6bdb62e3d7ed206bcaef0cd3499f9cb56dc1fb92
I've synced my branch with master, so you just need to pull it from my branch.

• Background of the project
  • already landed tens of standalone patches when I was in Oracle, all following descriptions are about patches not landed yet
  • A fat server is divided into several processing partitions (or cpu-partitions, CP as abbreviation), each partition contains: some cpu cores(or NUMA nodes) + memory pool + message queue + threads pool), it's a little like concept of virtual machine, although it's much simpler. It's not new thing, I just bring in this as a concept to replace what I used to call cpu_node + cpu-affinity, which is confusing because "node" is already used to present NUMA in linux kernel.
  • Although we still have one single namespace on MDS, but we can have several virtual processing partitions, lifetime of request should be localized on the processing partition as possible as we can, to reduce data/thread migration between CPUs, also reduce lock contentions and cacheline conflicts.
    • LND has message queue and threads-pool for each partition
    • LNet has EQ callback for each partition
    • Ptlrpc service has message queue and threads pool for each partition
  • number of CPU partitions is configurable (by new parameter "cpu_npartitions" of libcfs), libcfs will also automatically estimate "cpu_npartitions" based on number of CPUs. If we have cpu_node_num=N (N > 1), Lustre stack can have N standalone processing flows (unless they are contenting on the same resource), if we configured the system with cpu_npartitions=1, we have only one partition and lustre should act as current "master".
  • user can provide string pattern for cpu-partitions as well, i.e: "0[0, 2, 4, 6] 1[1, 3, 5, 7]", by this way, lustre will have two cpu-partitions, the first one contains core[0, 2, 4, 6], the second one presents core[1, 3, 5, 7], NB: those numbers inside bracket can be NUMA id as well.
  • number of cpu partitions < number of cores (or hyper-threadings), because modern computer can have hundreds or more cores, there are some major downsides if we have per-core schedulers(threads) pool: a) bad load balance especially on small/middle size cluster, b) too many threads overall;
    • on client/OSS/router, cpu_npartitions == NUMA node, dataflow of request/reply is localized on NUMA node
    • on MDS, a cpu partition will present a few cores(or hyper-threading), i.e: on system with 4 NUMA nodes and 64-cores, we have 8 cpu-partitions
    • we might hash different objects(by fid) to different cpu-partition in the future, it's an advanced feature which we don't have now
    • we can bind LNet NI on specified cpu partition for NUMIOA performance
  • these things helped a lot on performance of many target directories tests, but they can help nothing on shared directory performance at all, I never saw 20K+ file/sec in shared directory creatinon/removal tests (non-zero stripe), with 1.8, the number is < 7K files/sec
  • new pdirops patch works fine(LU-50), Andreas has already reviewed the prototype, now I have posted an the second version for review. The pdirops patch is different with the old one (we had 5 or 6 years ago?)
    • the old version is dynlock based, it has too many inline changes for ext4, and we probably need to change more to support N-level htree (large directory), we probably have to change ext4/namei.c and make it quite like our IAM.
    • the new version is htree_lock based, although implementation of the htree_lock is big & complex, but it just requires a few inline changes for ext4. htree_lock is more like a embedded component, it can be enabled/disabled easily, and need very few logic change to current ext4. The patch is big (about 2K lines), but only has about 200 LOC inline changes (including changes for N-level htree), and half of those inline changes are adding a new parameter to some functions.
    • htree_lock based pdirops patch have same performance as IAM dir, but without any interop issue
    • with pdirops patch on my branch, we can get 65K+ files/sec opencreate+close (narrow stripping file, aggregation performance on server) in my latest test on toro
  • MDD transaction scheduler will be totally dropped, If you still remember it, I mentioned previously that I wanted to add a relatively small threads pool in MDD to take over backend filesystem transactions, this thread pool size should be moderate and just enough for driving full throughput of metadata operations, this idea is majorly for share directory performance. However, pdirops patch works very good, so I will totally drop it (there is more detail in my mail to lustre-devel a few weeks ago)
• key component for landing
  • Tens of standalone patches
    • no dependency on each other, size of them are from a few LOC to a few hundreds LOC
    • some of them will be landed on 2.1, all of them will be landed on 2.2
  • pdirop patch + BH LRU size kernel patch
    • pdirop patch is big, but easy to maintain (at least it's easy to port it to rhel6)
    • I've sent another mail to Andreas and bzzz to explain why we need to increase size of BH LRU size, it will be a small patch
    • we would like to land them on 2.2 if we got enough resource or got funding on this
  • cpu-partition patches
    • it's the biggest chunk, includes several large patches spread over stack layers (libcfs, LNet, LND, ptlrpc server side, and some small changes to other modules like mdt, ost )
    • patches have dependencies.
    • The biggest challenge is inspection LNet + LNDs, isaac might give us some help to review, Lai Siyao will be the other inspector
    • if we got funding on this, we tend to land them on 2.2, otherwise it's more realistic for us to land them on 2.3

Can you post links to your patches here. Thank you.

http://review.whamcloud.com/#change,2346
http://review.whamcloud.com/#change,2461
http://review.whamcloud.com/#change,2523
http://review.whamcloud.com/#change,2558
http://review.whamcloud.com/#change,2638
http://review.whamcloud.com/#change,2718
http://review.whamcloud.com/#change,2725
http://review.whamcloud.com/#change,2729

FYI - They're still in review.

Re: the last two comments. Patches have landed. I'll copy an update of the patch list from Richard Henwood here:

c3e985e7e98f41ebf5ecb78887dcd2554601f7ef LU-56 ptlrpc: post rqbd with flag LNET_INS_LOCAL
8bbd62a7c0d2fc48d8f11e78d92bb42809968bba LU-56 ptlrpc: CPT affinity ptlrpc RS handlers
d800fc41a1abdaf7aaf6c0e3e7ddcdec489985a8 LU-56 ptlrpc: partitioned ptlrpc service
b43a6b1800265608cfa18159d4d0d006a1c23015 LU-56 o2iblnd: CPT affinity o2iblnd
82e02a17c0c645a8d156e51b8d8da5eaa68b8f5b LU-56 lnet: re-finalize failed ACK or routed message
1a73553d15b459208cbf7279ea6e5e5a110c632b LU-56 ksocklnd: CPT affinity socklnd
6b4b780895dfdeaca316862fbf1696983608f96d LU-56 lnet: wrong assertion for optimized GET
913c8e22cfc4fc5f52c4f0d6d3f0b4b86a7ac58c LU-56 lnet: tuning wildcard portals rotor
c03783fce46ae0b40db0680388df6e2d6fca5008 LU-56 lnet: SMP improvements for LNet selftest
7b2ab9beae02080797ff2da5105eaddadd67c151 LU-56 ldlm: SMP improvement for ldlm_lock_cancel
07b8db220e48782369f48d86213c5d404a628ded LU-56 ptlrpc: Reduce at_lock dance
c48a869557fe7663f4f3370b130d4c248958180e LU-56 libcfs: CPT affinity workitem scheduler
8a5b8dbda960b155f669c13602504f1233a84c7e LU-56 obdclass: SMP improvement for lu_key
e531dc437c56a08a65de9074a511faa55184712b LU-56 lnet: multiple cleanups for inspection
e069296630240947f1815e505067fd48033909f7 LU-56 lnet: allow user to bind NI on CPTs
a07e9d350b3e500c7be877f6dcf54380b86a9cbe LU-56 lnet: Partitioned LNet networks
5e1957841df3e771f3d72d8ea59180213430bbb9 LU-56 lnet: cleanup for rtrpool and LNet counter
279bbc81e03dc74d273ec12b4d9e703ca94404c4 LU-56 lnet: Partitioned LNet resources (ME/MD/EQ)
582c110231cf06bcd7e5e0b3bdf4f2058e18ebe4 LU-56 ptlrpc: cleanup of ptlrpc_unregister_service
ff0c89a73e141ce019ee2a94e5d01a8a37dd830a LU-56 ptlrpc: svc thread starting/stopping cleanup
25766da50b627648b04549ff3fb55af12acbcb4b LU-56 lnet: reduce stack usage of "match" functions
c7bff5640caff778d4cfca229672a2cc67b350d6 LU-56 lnet: Granulate LNet lock
24564b398f53009521aeda5d653e57fe8b525775 LU-56 ptlrpc: partition data for ptlrpc service
698d3088622b4610a84bd508f2b707a7a2dd1e3e LU-56 lnet: code cleanup for lib-move.c
38fcdd3966da09517ca176b962230b7dae43514c LU-56 lnet: match-table for Portals
f0aa1eef72e7438c2bd4b3eee821fefbc50d1f8e LU-56 lnet: code cleanup for lib-md.c
75a8f4b4aa9ad6bf697aedece539e62111e9029a LU-56 lnet: split lnet_commit_md and cleanup
06093c1f24da938418a0243259b5307c9fc338d5 LU-56 lnet: LNet message event cleanup
2118a8b92cec2df85d1bdbe2e58b389d83fe06b2 LU-56 lnet: eliminate a few locking dance in LNet
51a5b4df5bbbf5fd12c73d2722b230e93fe93327 LU-56 lnet: parse RC ping in event callback
b9bad9bd7d1c3271df916ee62091106e3f3c98b7 LU-56 lnet: router-checker (RC) cleanup
4fcc56be68c8c1667fbd91721d084874a2f05c3e LU-56 ptlrpc: common code to validate nthreads
ed22093b2d569fd0e93f35504580171114bf212d LU-56 lnet: move "match" functions to lib-ptl.c
a096d858b671f28fd4c5e6197b51643cd0780a50 LU-56 lnet: allow to create EQ with zero eq_size
c1366da8f43ecfb98ef3bdcf629eec8a2fc9cd4c LU-56 lnet: cleanup for LNet Event Queue
3211f6862cbbe96642db540e6593f3c614f9528c LU-56 lnet: new internal object lnet_peer_table
7a51ad347960ef2b9d1dfad14644c0bca35b80b6 LU-56 ptlrpc: clean up ptlrpc svc initializing APIs
facf5086667874c405c9ef6ce7f8f737868ffefd LU-56 lnet: container for LNet message
c3a57ec36441c75df03cfbec8f718e053aaad12a LU-56 lnet: abstract container for EQ/ME/MD
4bd9bf53728260d38efc74cac981318fe31280cd LU-56 lnet: add lnet_*_free_locked for LNet
c8da7bfbe0505175869973b25281b152940774b0 LU-56 libcfs: more common APIs in libcfs
b76f327de7836a854f204d28e61de52bc03011b1 LU-56 libcfs: export a few symbols from libcfs
3a92c850b094019e556577ec6cab5907538dcbf5 LU-56 libcfs: NUMA allocator and code cleanup
617e8e1229637908d4cce6725878dd5668960420 LU-56 libcfs: implementation of cpu partition
19ec037c0a9427250b87a69c53beb153d533ab1c LU-56 libcfs: move range expression parser to libcfs

Cray has been testing (a little) for functionality only. No significant issues to report so far.

SMP node affinity presentation for Lustre team Tech-call

SMP node affinity OpenSFS demonstration

Please let me know if more work is needed for this and I will reopen ticket.

patched mdtest which parses expression for multi-mount