[LU-1152] ls: cannot open directory .: Permission denied

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.7
• Type: Bug
• Priority: Major
• Reporter: Ben Evans (Inactive)
• Assignee: Zhenyu Xu
• Resolution: Fixed

Environment

Client: 1.8.7-wc1, 2.6.32-131.12.1.el6.x68_64
Server: 1.8.4

• Severity: 3
• Epic: client

Description

mount lustre on /mnt
ls /mnt (works)
cd /mnt/blah (works)
ls
ls: cannot open directory .: Permission denied
ls foo (works)
ls /mnt/blah (fails)
ls /mnt/blah/foo (works)

Running strace on ls, I get permission denied opening '.', regardless of directory permissions.

Comments

[Cliff White (Inactive)]

The user and group id's must be known the MDS, is the user in the MDS password files? Check the client and MDS system logs for errors.

[Ben Evans (Inactive)]

The user was root. I can run on an older Lustre/FS combo and it works.

I can do the following and it still fails:

mkdir /mnt/tmp
cd /mnt/tmp
ls

[Ben Evans (Inactive)]

sorry, I should clarify, an older Lustre Client/kernel version combo works fine.

also, in addition:

touch /mnt/tmp/foo
ls /mnt/tmp/foo

does work. I'm thinking there's some special handling of '.' that's causing an issue that I'm running into.

[Peter Jones]

Bobi

Could you please comment on this one?

Thanks

Peter

[Zhenyu Xu]

what's the output of "ls -l /mnt/"?

[Ben Evans (Inactive)]

ls -l /mnt displays everything correctly, but it is the only thing that does.

[Zhenyu Xu]

what's the permission/acl output of /mnt/blah ?

[Ben Evans (Inactive)]

drwxrwxrwx (I've tried many permutations around that theme, none work)

[Zhenyu Xu]

Would you mind enabling all debug message level "lctl set_param debug=-1" on client and servers, and collecting debug logs (client node as well as MDS server) during these operations? So that we can check out what the problem might be.

[Ben Evans (Inactive)]

Process for the dump:

mount client
start debugging
ls /mnt
ls -l /mnt
mkdir /mnt/testdir
ls -l /mnt/testdir (failed)
touch /mnt/testdir/testfile
ls -l /mnt/testdir (failed)
ls -l /mnt/testdir/testfile (success)
rm /mnt/testdir/testfile
rmdir /mnt/testdir (failed)
chmod 777 /mnt/testdir
rmdir /mnt/testdir (failed)

[Zhenyu Xu]

client request testdir's xattr

00000080:00200000:2:1330700770.421348:0:16553:0:(xattr.c:349:ll_getxattr()) VFS Op:inode=708444163/3303540295(ffff88010e00ed50), xattr security.selinux
...
00000002:00000001:2:1330700770.421799:0:16553:0:(mdc_request.c:367:mdc_xattr_common()) Process leaving via err_out (rc=18446744073709551555 : -61 : ffffffffffffffc3)
====>ENODATA

new client(1.8.7) old server(1.8.4) syndrome, I am afraid you'd update your server.

[Peter Piela (Inactive)]

Would disabling selinux be a workaround for this problem?

[Zhenyu Xu]

I think not, gotta use >= 1.8.7 server if your client is 1.8.7.

Since old MDS does not have the essential info (stripe info for client) as following MDS log shows.

$ grep fsfilt_ldiskfs_get_md /tmp/logs/1152/dump.mds.out
00002000:00000040:21:1330696880.700393:0:12708:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:9:1330696880.706343:0:12706:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112901473: rc -61
00002000:00000040:9:1330696880.707520:0:12710:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114144769: rc -61
00002000:00000040:15:1330696880.712199:0:12714:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112899589: rc -61
00002000:00000040:17:1330696880.713367:0:12723:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114275841: rc -61
00002000:00000040:19:1330696880.714589:0:31534:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112965121: rc -61
00002000:00000040:15:1330696880.715786:0:31557:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866819: rc -61
00002000:00000040:17:1330696880.716954:0:31555:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113817091: rc -61
00002000:00000040:1:1330696880.718103:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113882625: rc -61
00002000:00000040:23:1330696880.720901:0:12707:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866822: rc -61
00002000:00000040:22:1330696880.722087:0:31556:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113849857: rc -61
00002000:00000040:4:1330696939.175531:0:31531:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:17:1330696943.714265:0:31561:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:15:1330696943.715212:0:12720:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:7:1330696952.152840:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444164: rc -61
00002000:00000040:7:1330696952.152854:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330696957.721309:0:12711:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:3:1330696981.345390:0:12713:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:13:1330696991.321460:0:31535:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330697014.615160:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61

[Ben Evans (Inactive)]

disabling selinux is a workaround, but not a fix.

Since 1.8.7-wc1 clients work on older versions of Linux with the same filesystem, this looks like a kernel incompatibility, rather than a client/server issue (or one that Lustre aggravates).

[Xuezhao Liu]

First, the selinux should be disabled.

Secondly, for "fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61" such log, in the case that only 1 OST installed on the server, is it possible the EA(stripe info) be empty? Seems it is just a kind of warning/log message, not really error.

[Zhenyu Xu]

Ben,

Can you describe more info about the step? It's 1.8.4 Lustre server, and all server devices were formated with this 1.8.4 Lustre server? What Linux kernel version do these servers use? What steps did you take? Just mounted the Lustre system and "mkdir /mnt/testdir" and then "ls /mnt/testdir" failed?

[Ben Evans (Inactive)]

server kernel version is 2.6.18-194.3.1.0.1.el5 and Lustre 1.8.4
Formatted with that kernel/Lustre version.
8 OSTs, default striping.

Steps on the client are exactly as you describe, mount, mkdir, ls.

Disabling selinux on the client fixed the ls issue.

[Zhenyu Xu]

hmm, strangely, I have tried "SELINUX=enforcing" and "SELINUX=targeted", and cannot hit it. Anyway, Lustre should not work with selinux enabled.

[Peter Jones]

As per Terascala - this ticket can be closed