[LU-1234] Executing binary stored on Lustre results in " (deleted)" appended to /proc/self/exec

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Prakash Surya (Inactive)

Description

We have reports that when running an executable stored on Lustre, the symlink /proc/self/exec will have " (deleted)" appended to it. This can be normal behavior under Linux, if the executable has in fact been overwritten or deleted. But I have been told this is not the case, and I would not be surprised if there is a Lustre bug here. Digging into the kernel sources, it appears the following snippets are relevant:

__d_path

 if (d_unlinked(dentry) &&
         (prepend(&end, &buflen, " (deleted)", 10) != 0))
                 goto Elong;

static inline int d_unhashed(struct dentry *dentry)
{
        return (dentry->d_flags & DCACHE_UNHASHED);
}

static inline int d_unlinked(struct dentry *dentry)
{
        return d_unhashed(dentry) && !IS_ROOT(dentry);
}

So, the string " (deleted)" is appended when the dentry in question has the DCACHE_UNHASHED flag set. Perhaps there's a situation where Lustre fails to take a reference on the dentry, allowing it to be dropped from the cache when it shouldn't be?

Also note, I haven't been able to reproduce this issue so it's been difficult to debug, or even determine exactly where the issue lies.

Comments

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el5,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/llite/file.c
• lustre/include/linux/lustre_patchless_compat.h
• lustre/llite/llite_internal.h
• lustre/llite/namei.c
• lustre/llite/statahead.c
• lustre/llite/dcache.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,server,el5,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/include/linux/lustre_patchless_compat.h
• lustre/llite/dcache.c
• lustre/llite/namei.c
• lustre/llite/llite_internal.h
• lustre/llite/statahead.c
• lustre/include/linux/lustre_compat25.h
• lustre/llite/file.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el6,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/llite/llite_internal.h
• lustre/llite/file.c
• lustre/llite/namei.c
• lustre/include/linux/lustre_patchless_compat.h
• lustre/include/linux/lustre_compat25.h
• lustre/llite/dcache.c
• lustre/llite/statahead.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el5,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/llite/file.c
• lustre/llite/dcache.c
• lustre/llite/namei.c
• lustre/llite/statahead.c
• lustre/llite/llite_internal.h
• lustre/include/linux/lustre_patchless_compat.h

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el6,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/llite/namei.c
• lustre/llite/dcache.c
• lustre/include/linux/lustre_compat25.h
• lustre/include/linux/lustre_patchless_compat.h
• lustre/llite/file.c
• lustre/llite/statahead.c
• lustre/llite/llite_internal.h

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el5,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/llite/file.c
• lustre/llite/llite_internal.h
• lustre/llite/statahead.c
• lustre/llite/namei.c
• lustre/include/linux/lustre_compat25.h
• lustre/llite/dcache.c
• lustre/include/linux/lustre_patchless_compat.h

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el6,inkernel #4594
LU-1234 dcache: don't drop invalid dentry arbitrarily (Revision c7619a3c9c0b5ddc040612169402f6ab75bb3a49)

Result = SUCCESS
Lai Siyao : c7619a3c9c0b5ddc040612169402f6ab75bb3a49
Files :

• lustre/llite/statahead.c
• lustre/llite/llite_internal.h
• lustre/llite/namei.c
• lustre/llite/dcache.c
• lustre/include/linux/lustre_patchless_compat.h
• lustre/llite/file.c
• lustre/include/linux/lustre_compat25.h

[Lai Siyao]

Review is on http://review.whamcloud.com/#change,2400.

[Christopher Morrone (Inactive)]

If the user still has this executable available, it should be possible to compare "ls -li" and/or "lfs path2fid" and/or "lfs getstripe -v" for the deleted binary with the existing binary to see if they are in fact the same file.

How do we get a handle to the "deleted binary" on which to run "ls -li" or anything else if it has been deleted?

We could perhaps use lsof to get the inode of the in-use binary and compare it to the current binary's inode using "ls -li".

But I am fairly certain that the file has not been deleted. The " (deleted)" extension disappears after time for the same running binary, which should definitely not happen if the file was legitimately deleted or overwritten.

[Christopher Morrone (Inactive)]

The linux kernel uses d_unlinked() this way, not an application.

The application impact comes when a user tries to use a user-space debugger, but the debugger will not work because the executable name pointed to by /proc/<pid>/exe contains the string " (deleted)".

Please do make a fix for 2.1.

[Lai Siyao]

Actually DCACHE_LUSTRE_INVALID is redefined in llite_internal.h if it's not in kernel header file. And in the future it's planned to move into struct ll_dentry_data to avoid conflict with kernel flags.

Yes, with http://review.whamcloud.com/#change,1865, referenced dentry will never be dropped if the file is not unlinked or renamed.

Could you explain a bit on why your application rely on this? IMHO d_unlinked() normally makes sense for local filesystem, but for network filesystem, it often means dentry is invalid and dropped from dcache, and client needs lookup from server again (eg. NFS has such semantic). However with the patch mentioned, Lustre can act the same as local filesystem, though the original intention of the patch is not for this. If you do need this, I can help port this patch to 2.1.