[LU-10892] hang at 'echo clear > /proc/fs/lustre/ldlm/namespaces/.../lru_size'

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.5.5
• Type: Bug
• Priority: Minor
• Reporter: Olaf Faaland
• Assignee: Peter Jones
• Resolution: Fixed
• Labels: llnl
• Severity: 3

Description

We are encountering frequent hangs when we execute:

echo clear > $server/lru_size

where $server is a path like /proc/fs/lustre/ldlm/namespaces/ls6-OST000a-osc-<UUID>/.

In the cases we've documented the target is an OST.   That OST shows as active in lfs check servers.  We see no indication of problems (on the OST (nothing in console logs, no flapping connections, etc.).

The stack trace looks like this.

__ldlm_bl_to_thread+0x144
ldlm_bl_to_thread+0x473
ldlm_bl_to_thread_list+0x19
ldlm_cancel_lru+0x70
lprocfs_lru_size_seq_write+0x10c
proc_reg_write+0x7e
...

The client version is lustre-2.5.5-11chaos. The server version is lustre 2.8.2.

Code where stuck thread is blocking:

(gdb) l *(__ldlm_bl_to_thread+0x144)
0x28874 is in __ldlm_bl_to_thread (/usr/src/debug/lustre-2.5.5/lustre/ldlm/ldlm_lockd.c:1997).
1992 wake_up(&blp->blp_waitq);
1993
1994 /* can not check blwi->blwi_flags as blwi could be already freed in
1995 LCF_ASYNC mode */
1996 if (!(cancel_flags & LCF_ASYNC))
1997         wait_for_completion(&blwi->blwi_comp);
1998
1999 RETURN(0);
2000 }
2001
(gdb) quit

 
We find that we can kill the user space process without any obvious ill effects. The user space process dies.

We are working on retiring our Lustre 2.5 systems, so a workaround is sufficient. Our questions are:
1. Is it correct that the locks being purged are not protecting any dirty cache on the client?
2. Can simply kill these stuck processes without data loss?

Comments

[Andreas Dilger]

Olaf,
the locks in the LRU may be protecting clean or dirty data pages (and metadata for MDC locks), but the locks are not cancelled until all of the pages covered by the lock have been written and dropped from cache. The "echo clear > ...<target>/lru_size" (or preferably "lctl set_param osc.<target>.lru_size=clear" so that your scripts will not break as these files are moved from /proc to /sys) will drop all unused locks on the client (for the specific target, if specified or all targets if '<target>=*' is used). The only locks not dropped will be those currently in use by an active system call.

Killing the operation will, at worst, mean that some of the DLM locks may not be cancelled immediately, and will be expired by some other mechanism (age, number of locks, server load).

[Olaf Faaland]

Thank you.