[LU-12429] Single client buffered SSF write is slower than O_DIRECT

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.13.0
• Severity: 3
• Assignee: Dongyang Li
• Reporter: Shuichi Ihara

Description

Single client's SSF doesn't scale by nubmer of process

# mpirun --allow-run-as-root -np X /work/tools/bin/ior -w -t 16m -b $((32/X))g -e -o file

NP     Write(MB/s)
  1     1594
  2     2525
  4     1892
  8     2032
 16     1812

A framegraph output at ior with NP=16 pointed out huge amount cost of spin_lock in add_to_page_cache_lru() and set_page_dirty(). At the resutls, Buffered SSF write on single client is slower than SSF with O_DIRECT. Here is my quick test resutls of single client SSF with/without O_DIRECT.

# mpirun -np 16 --allow-run-as-root /work/tools/bin/ior -w -t 16m -b 4g -e -o /scratch0/stripe/file 
Max Write: 1806.31 MiB/sec (1894.06 MB/sec)

# mpirun -np 16 --allow-run-as-root /work/tools/bin/ior -w -t 16m -b 4g -e -o /scratch0/stripe/file -B
Max Write: 5547.13 MiB/sec (5816.58 MB/sec)

Issue Links

• is related to: LU-247 Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO
• is related to: LU-9920 Use pagevec for marking pages dirty