[LU-9210] 'ls' hung because of statahead delay

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Lai Siyao

Description

Our main problem is that sometimes ls -l takes a long time to answer as shown below.

[sh-5-34 ~]$ alias ls
alias ls='ls --color=auto'

[sh-5-34 ~]$ echo $GROUP
/groups/alice

Sometimes ls takes 30 seconds to 1 minute:

[sthiell@sh-5-34 ~]$ time ls $GROUP
addiso feel_free_to_test foobar pictures preese-dir software some_group_datadir vmd-1.9.2
real 0m30.012s
user 0m0.001s
sys 0m0.004s

Note 1: when I enable lustre debugging on the client, I cannot reproduce the issue, which is a bit annoying for the bug report.
Note 2: when I set statahead_max to 0 the problem is gone, and is easily seen as soon as I re-enable statahead_max (set to 1 or more)
Note 3: it’s also very difficult to reproduce the issue when using ls with strace, but I have seen one time that ls was blocking in lstat()

The relevant part of the log is as below:

00000080:00400000:3.0:1489184551.164004:0:25907:0:(statahead.c:683:ll_statahead_interpret()) sa_entry software rc -13
00000080:00400000:0.0:1489184581.163471:0:25086:0:(statahead.c:1666:ll_statahead()) revalidate statahead software: -11.

Obviously statahead failure didn't notify 'ls' process in time, and cause it slow.

Issue Links

• is related to: LU-11616 Optimize handling statahead delay

Comments

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27330/
Subject: LU-9210 statahead: missing barrier before wake_up
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d7fe5b6152d9f01f003c93cc1367455c30dc35ed

[Gerrit Updater]

Bob Glossman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27330
Subject: LU-9210 statahead: missing barrier before wake_up
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1536a7b780d77fcfd11ce8e81b1ca4316a7040ad

[Gerrit Updater]

Bob Glossman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27329
Subject: LU-9210 statahead: missing barrier before wake_up
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a291004810c0f17f0410da80238d82127b78d8d8