[LU-3769] compile bench real slow

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Minor
• Reporter: Gabriele Paciucci (Inactive)
• Assignee: WC Triage
• Resolution: Won't Fix
• Epic/Theme: Performance
• Severity: 3

Description

server: lustre-2.4.0-2.6.32_358.11.1.el6_lustre.gef18aa1.x86_64_ga5e72ed.x86_64 on CENTOS 6.4
client: lustre-client-2.4.52-3.9.9_302.fc19.x86_64_gfdd4844.x86_64 on FC19

the client is mounted with flock parameter:
192.168.2.104@o2ib:/lustrefs on /lustre type lustre (rw,flock)

I have run some micro-benchmark like mdstat/iozone/IOR without problem with good results.
Customer asks me to run compilebench benchmark. I have downloaded the compilebench rpm from http://build.whamcloud.com/job/toolkit/

Comments

[Gabriele Paciucci (Inactive)]

I have set this parameters on the client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_rpcs_in_flight=256

Compilebench results against a local disk:
compilebench -D /tmp/ -i 2 -r 2 --makej
using working directory /tmp/, 2 intial dirs 2 runs
native unpatched native-0 222MB in 1.00 seconds (222.37 MB/s)
native patched native-0 109MB in 0.31 seconds (353.78 MB/s)
native patched compiled native-0 691MB in 0.51 seconds (1356.07 MB/s)
create dir kernel-0 222MB in 3.06 seconds (72.67 MB/s)
create dir kernel-1 222MB in 3.14 seconds (70.82 MB/s)
compile dir kernel-1 680MB in 0.89 seconds (764.76 MB/s)
compile dir kernel-0 680MB in 0.61 seconds (1115.79 MB/s)
read dir kernel-1 in 21.46 52.44 MB/s
read dir kernel-0 in 20.26 55.55 MB/s
read dir kernel-1 in 0.66 1705.12 MB/s
delete kernel-1 in 1.36 seconds
delete kernel-0 in 1.04 seconds

run complete:
==========================================================================
intial create total runs 2 avg 71.75 MB/s (user 0.40s sys 0.79s)
no runs for create
no runs for patch
compile total runs 2 avg 940.28 MB/s (user 0.08s sys 0.50s)
no runs for clean
no runs for read tree
read compiled tree total runs 3 avg 604.37 MB/s (user 0.56s sys 1.16s)
no runs for delete tree
delete compiled tree total runs 2 avg 1.20 seconds (user 0.18s sys 0.35s)
no runs for stat tree
no runs for stat compiled tree

Compilebench results on the same machine against the lustre fs:
compilebench -D /lustre -i 2 -r 2 --makej
using working directory /lustre/GABRIELE/, 2 intial dirs 2 runs
native unpatched native-0 222MB in 37.65 seconds (5.91 MB/s)
native patched native-0 109MB in 10.89 seconds (10.07 MB/s)
native patched compiled native-0 691MB in 5.40 seconds (128.07 MB/s)
create dir kernel-0 222MB in 36.36 seconds (6.12 MB/s)
create dir kernel-1 222MB in 37.58 seconds (5.92 MB/s)
compile dir kernel-1 680MB in 6.94 seconds (98.07 MB/s)
compile dir kernel-0 680MB in 6.61 seconds (102.97 MB/s)
read dir kernel-1 in 32.43 27.84 MB/s
read dir kernel-0 in 31.67 28.51 MB/s
read dir kernel-1 in 3.96 228.03 MB/s

the delete process was too long to complete....

[Oleg Drokin]

So is it slow compared to before, or is it slo because it was always slow on lustre (and lustre is not very good at these workloads).

(note we run compile bench regularly in testing so you can track history of performance on those).

[Gabriele Paciucci (Inactive)]

Hi Oleg,
I have no experience with compilebench... Could you please give some suggestions how to run in the right way this tool? I have found benefits avoiding sync and drop cache (-n).

[Oleg Drokin]

You probably need to ask somebody like Yu Jian for advice on this.

[Gabriele Paciucci (Inactive)]

Thanks Oleg