[LU-9210] 'ls' hung because of statahead delay

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Lai Siyao

Description

Our main problem is that sometimes ls -l takes a long time to answer as shown below.

[sh-5-34 ~]$ alias ls
alias ls='ls --color=auto'

[sh-5-34 ~]$ echo $GROUP
/groups/alice

Sometimes ls takes 30 seconds to 1 minute:

[sthiell@sh-5-34 ~]$ time ls $GROUP
addiso feel_free_to_test foobar pictures preese-dir software some_group_datadir vmd-1.9.2
real 0m30.012s
user 0m0.001s
sys 0m0.004s

Note 1: when I enable lustre debugging on the client, I cannot reproduce the issue, which is a bit annoying for the bug report.
Note 2: when I set statahead_max to 0 the problem is gone, and is easily seen as soon as I re-enable statahead_max (set to 1 or more)
Note 3: it’s also very difficult to reproduce the issue when using ls with strace, but I have seen one time that ls was blocking in lstat()

The relevant part of the log is as below:

00000080:00400000:3.0:1489184551.164004:0:25907:0:(statahead.c:683:ll_statahead_interpret()) sa_entry software rc -13
00000080:00400000:0.0:1489184581.163471:0:25086:0:(statahead.c:1666:ll_statahead()) revalidate statahead software: -11.

Obviously statahead failure didn't notify 'ls' process in time, and cause it slow.

Issue Links

• is related to: LU-11616 Optimize handling statahead delay