[LU-4498] MDT thread hung, ls fails on directory

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.6
• Type: Bug
• Priority: Major
• Reporter: Kit Westneat (Inactive)
• Assignee: Zhenyu Xu
• Resolution: Duplicate
• Severity: 3

Description

IU is running into an issue where running ls on a certain file causes clients to get evicted. It appears as if there is a hung MDT thread holding a lock on the file. After the MDT is rebooted, listing the directory and file works fine.

We were able to capture client debug logs and a backtrace of all the threads from the running system, but due to an issue with STONITH, we were unable to get a good vmcore from the system. Also when we tried to get debug logs from the MDT, the log overflowed, even with a 20GB buffer.

We are currently waiting for the issue to reappear and will get debug logs on a quiesced system, as well as a good vmcore.

I'll upload the logs we have. Is there anything else we should be looking to get?

Comments

[Kit Westneat (Inactive)]

Here's an example of the directory listing. I forgot to mention it takes hours for the MDT to actually evict the client.

ls -al
/N/dc2/scratch/kmoriya/bggen_8.4_9.0-2013-12-11/0000/000100
—
Begin 2013-12-20_13:37:32
total 164
drwxr-xr-x 34 1012412 401 4096 Dec 11 20:59 .
drwxr-xr-x 202 1012412 401 36864 Dec 16 09:32 ..
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:23 00003200
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003201
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003202
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003203
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003204
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003205
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003206
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003207
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003208
?--------- ? ? ? ? ? 00003209
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003210
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003211
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003212
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003213
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003214
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003215
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003216
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003217
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003218
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003219
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003220
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003221
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003222
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003223
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003224
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003225
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003226
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003227
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003228
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003229
drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003230
drwxr-xr-x 2 1012412 401 4096 Dec 12 07:42 00003231
End 2013-12-20_15:44:49

[Cliff White (Inactive)]

Is there anything else you can tell us about the 'certain file' ?
Is it always the same file, or type of file?
Have you run fsck on the MDT lately?

[Kit Westneat (Inactive)]

It's happened three times on seemingly unrelated files. It looks like the last e2fsck was on June 12. FWIW there don't seem to be any ldisk errors in the logs anytime recently.

[Peter Jones]

Bobijam

Could you please help with this one?

Thanks

Peter

[Zhenyu Xu]

Fanyong,

does it look like dir hash collision issue?

[nasf (Inactive)]

According to the client side log, every getattr RPC is for different file, so it cannot to say that the "ls" fall into hash collision. On the other hand, the MDT side shows that during the "ls" processing, there are some "open-create" operations. What they are for?

[Zhenyu Xu]

the client log shows that ls happened during 2014/01/09 23:46:11 to 2014/01/09 23:46:12, which took 1 seconds and does not match to the 1st comment report (from 2013-12-20_13:37:32 to 2013-12-20_15:44:49). Do you have logs which covers the issue time span?

[Zhenyu Xu]

would you mind trying this patch http://review.whamcloud.com/8936 , it is a backport of dir hash collision fix patch.

[Peter Jones]

Bobijam

Could this be related to LU-4616, the root cause of which has now been established?

Peter

[Zhenyu Xu]

yes, I think it's related to LU-4616

[Peter Jones]

ok so then let's mark it as a duplicate of LU-4616 and reopen or open a new ticket if it manifests itself again now that LU-4616 seems to have been addressed.