[#LU-1234] Executing binary stored on Lustre results in " (deleted)" appended to /proc/self/exec

[LU-1234] Executing binary stored on Lustre results in " (deleted)" appended to /proc/self/exec Created: 19/Mar/12 Updated: 04/Mar/22 Resolved: 31/May/13
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None

Type:

Bug

Priority:

Minor

Reporter:

Prakash Surya (Inactive)

Assignee:

Lai Siyao

Resolution:

Fixed

Votes:

Labels:

None

Severity:

Rank (Obsolete):

8520

Description

We have reports that when running an executable stored on Lustre, the symlink /proc/self/exec will have " (deleted)" appended to it. This can be normal behavior under Linux, if the executable has in fact been overwritten or deleted. But I have been told this is not the case, and I would not be surprised if there is a Lustre bug here. Digging into the kernel sources, it appears the following snippets are relevant:

__d_path

 if (d_unlinked(dentry) &&
         (prepend(&end, &buflen, " (deleted)", 10) != 0))
                 goto Elong;

static inline int d_unhashed(struct dentry *dentry)
{
        return (dentry->d_flags & DCACHE_UNHASHED);
}

static inline int d_unlinked(struct dentry *dentry)
{
        return d_unhashed(dentry) && !IS_ROOT(dentry);
}

So, the string " (deleted)" is appended when the dentry in question has the DCACHE_UNHASHED flag set. Perhaps there's a situation where Lustre fails to take a reference on the dentry, allowing it to be dropped from the cache when it shouldn't be?

Also note, I haven't been able to reproduce this issue so it's been difficult to debug, or even determine exactly where the issue lies.

Comments

Comment by Peter Jones [ 19/Mar/12 ]

Lai

Could you please comment on this one?

Thanks

Peter

Comment by Andreas Dilger [ 19/Mar/12 ]

If the user still has this executable available, it should be possible to compare "ls -li" and/or "lfs path2fid" and/or "lfs getstripe -v" for the deleted binary with the existing binary to see if they are in fact the same file.

It is possible that the user began running this executable, then compiled a new version, or something similar, which replaced the old binary in-place.

Comment by Prakash Surya (Inactive) [ 19/Mar/12 ]

If the user still has this executable available, it should be possible to compare "ls -li" and/or "lfs path2fid" and/or "lfs getstripe -v" for the deleted binary with the existing binary to see if they are in fact the same file.

I'll pass this information along.

It is possible that the user began running this executable, then compiled a new version, or something similar, which replaced the old binary in-place.

Well I haven't verified this myself, but we've been assured this isn't happening. I believe Chris has specifically asked about this case.

I believe the " (deleted)" comes and goes sporadically as they poke around on the system. Essentially, the user kicks off a job, realizes it's taking longer than expected to start, and then tries to attach a debugger to it. The debugger in turn doesn't understand the " (deleted)" portion. I believe at this point he/she might log in and poke around looking at the /proc entries of the job and running other commands, perhaps "ls"-ing the file.

I'm not sure if any of that is helpful, but I thought that if it is an issue of the dentry being dropped (possibly due to memory pressure? I don't have much info so I'm just making an assumption), "ls"-ing the file would bring the dentry back. And thus, causing " (deleted)" to disappear briefly, only to reappear once the dentry is dropped again.

Comment by Lai Siyao [ 20/Mar/12 ]

Lustre tends to drop invalid dentry (ie. dentry without LOOKUP ldlm lock held on client) from dcache, and it's safe to drop dentry even it's referenced, it's just that others won't find this dentry from dcache, and needs lookup it.

But this is not necessary, because there is already a flag DCACHE_LUSTRE_INVALID to mark a dentry invalid, there is a patch for 'kernel dcache scalability changes': http://review.whamcloud.com/#change,1865, which won't often drop invalid dentries any more (but discard it at the put of last refcount), Prakash, do you have a chance to verify that works (If this executable won't stop immediately, you can chgrp of this file on another client, which will revoke the lookup lock on the first client)?

Comment by Prakash Surya (Inactive) [ 20/Mar/12 ]

Lai, Thanks for the help. Our only "reproducer" (not even sure how reliable it is) at the moment is to run this user's large (~1024 node) job in production. So, it might be a little difficult to test that patch out, but I'll talk to Chris and see what we can manage.

Comment by Christopher Morrone [ 21/Mar/12 ]

Whoops! Our kernel patch that adds DCACHE_LUSTRE_INVALID has fallen behind, and despite applying cleanly makes DCACHE_LUSTRE_INVALID have the same value as DCACHE_COOKIE. That probably isn't helping.

Comment by Christopher Morrone [ 21/Mar/12 ]

It looks to me like the only user of dcookies is oprofile, so I don't think that the conflict with DCACHE_COOKIE is implicated here. But I have fixed it in out next kernel.

Comment by Christopher Morrone [ 21/Mar/12 ]

The kernel seems to think that a running executable will have a cached dentry associated with it. In this case, it does NOT appear to be safe to drop the dentry. So rather than making it less likely that the dentry will be dropped, we need to make absolutely sure that we don't drop the dentry as long as the executable remains in use and unchanged.

Will http://review.whamcloud.com/#change,1865 do that? If so, we'll need it backported to 2.1.

Also, I am a little concerned about relying on DCACHE_LUSTRE_INVALID for a solution. Lustre on the client side is already supposed to work with a patchless kernel, so relying on a kernel patch for correctness is a problem.

Comment by Lai Siyao [ 21/Mar/12 ]

Actually DCACHE_LUSTRE_INVALID is redefined in llite_internal.h if it's not in kernel header file. And in the future it's planned to move into struct ll_dentry_data to avoid conflict with kernel flags.

Yes, with http://review.whamcloud.com/#change,1865, referenced dentry will never be dropped if the file is not unlinked or renamed.

Could you explain a bit on why your application rely on this? IMHO d_unlinked() normally makes sense for local filesystem, but for network filesystem, it often means dentry is invalid and dropped from dcache, and client needs lookup from server again (eg. NFS has such semantic). However with the patch mentioned, Lustre can act the same as local filesystem, though the original intention of the patch is not for this. If you do need this, I can help port this patch to 2.1.

Comment by Christopher Morrone [ 23/Mar/12 ]

The linux kernel uses d_unlinked() this way, not an application.

The application impact comes when a user tries to use a user-space debugger, but the debugger will not work because the executable name pointed to by /proc/<pid>/exe contains the string " (deleted)".

Please do make a fix for 2.1.

Comment by Christopher Morrone [ 26/Mar/12 ]

If the user still has this executable available, it should be possible to compare "ls -li" and/or "lfs path2fid" and/or "lfs getstripe -v" for the deleted binary with the existing binary to see if they are in fact the same file.

How do we get a handle to the "deleted binary" on which to run "ls -li" or anything else if it has been deleted?

We could perhaps use lsof to get the inode of the in-use binary and compare it to the current binary's inode using "ls -li".

But I am fairly certain that the file has not been deleted. The " (deleted)" extension disappears after time for the same running binary, which should definitely not happen if the file was legitimately deleted or overwritten.

Comment by Lai Siyao [ 28/Mar/12 ]

Review is on http://review.whamcloud.com/#change,2400.