[LU-983] On a 1.8.6 client/host, tar is performing 10K reads, and the RPCs are typically one page in size for small files==tar is slow for small files using 1.8.6 vs 1.8.5

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.2, Lustre 1.8.8
• Affects Version/s: Lustre 1.8.6
• Labels: performance
• Environment: cent 5.5/5.6 on lustre servers, sles11sp1 client (both ofed 1.5.2 and 1.5.3.1 have been tested on sles11sp1 clients)
• Severity: 3

Epic

client performance

• Assignee: Di Wang
• Reporter: James Karellas (Inactive)

Description

This is all with respect to small files. Large files seem to be ok.

We have been focusing on tar, but suspect the problem is with small
file read performance in general on 1.8.6 clients. We are aware that
WC has a version of tar (LU-682), which we tried but it didn't help at all.

On a 1.8.5 host, tar is performing 10K reads, and the RPCs
are typically falling into the 128 page bin of the histogram. Tar performs
as expected.

On a 1.8.6 host, tar is performing 10K reads, and the RPCs are typically
one page in size. That's bad.

We are having some problems with "collectl" on 1.8.6 filesystems or we'd
have better data for you.

As far as real world issues, we have several complaints form user that are
doing small IO. Jay lan reported this to wc-discuss but we havne't seen
antyhing come across yet. Here is what he reported:

Our users reported a serious performance issue in 1.8.6. The time
needed to tar a directory of 14k files (total size 6.3G, tar file
is stripe size 30) is ~6 minutes in lustre1.8.5, but ~120 minutes
using 1.8.6. Our nas-1.8.6 is very close to 1.8.6-wc release, but
our nas-1.8.5 was based on LLNL's version.

Is there a known issue on tar'ing large number of small files in
1.8.6? I am aware of the lustre-tar and downloaded the rpm
from the Whamcloud site for our admin. That version does
not seem to help.

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA
• Trackbacks: Changelog 1.8 Changes from version 1.8.7wc1 to version 1.8.8wc1 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.13.1.el6 (RHEL6) Recommended e2fsprogs version: 1.41.90....
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....