[LU-7828] Traversal directory is very slow

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: nasf (Inactive)

Description

Some customer found that:
Several customers said "ls" is slow with latest Lustre codes.
We tried to figure out detail and the following codes are easy to reproudce at the cutomer site. But, if we changed "ls -F" with "ls -F *", the problem didn't reproduce.

#!/bin/bash

for ((i=0;i<2583;i++)); do 
        dd if=/dev/zero of=file$i.dat bs=1024 count=400 > /dev/null 2>&1 
done

for ((i=0;i<20;i++)); do
        (time ls -F > /dev/null) >> lscheck.txt
done

rm -f file*.dat
echo ""
cat lscheck.txt
rm -f lscheck.txt
exit

Above test is just repeating "ls -F" 10 times after 2583 file creation.
Please see below. The most of cases finish quickly (less than 1 sec), but sevral cases, it took 30 sec.
And, one of strange thing, alltime it takes 30 sec or 30 sec x N.

This is 100% reproducable and we trying strace or ftrace to find where is most highest cost of system call.
However, when if we run "strace -f ls -F", the problem never reproduced which is also strange.

But, the problem only reproduced on the Lustre..