[LU-513] cfs_wait_event_interruptible_exclusive is not exclusive

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.1
• Environment: PPC64, RHEL6.1 (but should apply to all platforms)
• Severity: 3
• Assignee: Liang Zhen (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

It looks like cfs_wait_event_interruptible_exclusive() was accidentally written to use wait_event_interruptible(), when it should use wait_event_interruptible_exclusive().

This was expecially apparent when trying to run lnet selftest on Sequoia IO Nodes where there is 4-way SMT with 17 cores. This makes 68 "cpus" in linux, so Lustre is creating 68 cfs_wi_sd* scheduler threads.

Even when writing to a single peer (over o2iblnd) with concurrency 1, I saw ALL of the cfs_wi_sd threads eating all of the CPU time. Throughput numbers were also quite low (680MB/s writing to an x86_64 server, concurrency level 16). My suspicion was that all of the tasks were being woken up instead of only one for each work item. That is what eventually made me find the missing _exclusive() extension.

When I changed the it to use the _exclusive() version of the function, the cfs_wi_sd* threads no longer appear at the top of "top"'s display, and throughput jumped to 2750MB/s on the write test.

This would be a problem on all architectures, but with its 68 "processors", it is just more obviously a problem on Sequoia.

I'll submit a one-line patch.

Issue Links

• Trackbacks: Changelog 2.1 Changes from version 2.1.0 to version 2.1.1 Server support for kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) 2.6.32.360....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....