[LU-1152] ls: cannot open directory .: Permission denied

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 1.8.7
• Environment: Client: 1.8.7-wc1, 2.6.32-131.12.1.el6.x68_64; Server: 1.8.4
• Severity: 3

Epic

client

• Assignee: Zhenyu Xu
• Reporter: Ben Evans (Inactive)

Description

mount lustre on /mnt
ls /mnt (works)
cd /mnt/blah (works)
ls
ls: cannot open directory .: Permission denied
ls foo (works)
ls /mnt/blah (fails)
ls /mnt/blah/foo (works)

Running strace on ls, I get permission denied opening '.', regardless of directory permissions.

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA

Comments

[Ben Evans (Inactive)]

disabling selinux is a workaround, but not a fix.

Since 1.8.7-wc1 clients work on older versions of Linux with the same filesystem, this looks like a kernel incompatibility, rather than a client/server issue (or one that Lustre aggravates).

[Zhenyu Xu]

I think not, gotta use >= 1.8.7 server if your client is 1.8.7.

Since old MDS does not have the essential info (stripe info for client) as following MDS log shows.

$ grep fsfilt_ldiskfs_get_md /tmp/logs/1152/dump.mds.out
00002000:00000040:21:1330696880.700393:0:12708:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:9:1330696880.706343:0:12706:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112901473: rc -61
00002000:00000040:9:1330696880.707520:0:12710:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114144769: rc -61
00002000:00000040:15:1330696880.712199:0:12714:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112899589: rc -61
00002000:00000040:17:1330696880.713367:0:12723:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114275841: rc -61
00002000:00000040:19:1330696880.714589:0:31534:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112965121: rc -61
00002000:00000040:15:1330696880.715786:0:31557:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866819: rc -61
00002000:00000040:17:1330696880.716954:0:31555:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113817091: rc -61
00002000:00000040:1:1330696880.718103:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113882625: rc -61
00002000:00000040:23:1330696880.720901:0:12707:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866822: rc -61
00002000:00000040:22:1330696880.722087:0:31556:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113849857: rc -61
00002000:00000040:4:1330696939.175531:0:31531:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:17:1330696943.714265:0:31561:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:15:1330696943.715212:0:12720:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:7:1330696952.152840:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444164: rc -61
00002000:00000040:7:1330696952.152854:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330696957.721309:0:12711:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:3:1330696981.345390:0:12713:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:13:1330696991.321460:0:31535:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330697014.615160:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61

[Peter Piela (Inactive)]

Would disabling selinux be a workaround for this problem?

[Zhenyu Xu]

client request testdir's xattr

00000080:00200000:2:1330700770.421348:0:16553:0:(xattr.c:349:ll_getxattr()) VFS Op:inode=708444163/3303540295(ffff88010e00ed50), xattr security.selinux
...
00000002:00000001:2:1330700770.421799:0:16553:0:(mdc_request.c:367:mdc_xattr_common()) Process leaving via err_out (rc=18446744073709551555 : -61 : ffffffffffffffc3)
====>ENODATA

new client(1.8.7) old server(1.8.4) syndrome, I am afraid you'd update your server.

[Ben Evans (Inactive)]

Process for the dump:

mount client
start debugging
ls /mnt
ls -l /mnt
mkdir /mnt/testdir
ls -l /mnt/testdir (failed)
touch /mnt/testdir/testfile
ls -l /mnt/testdir (failed)
ls -l /mnt/testdir/testfile (success)
rm /mnt/testdir/testfile
rmdir /mnt/testdir (failed)
chmod 777 /mnt/testdir
rmdir /mnt/testdir (failed)

[Zhenyu Xu]

Would you mind enabling all debug message level "lctl set_param debug=-1" on client and servers, and collecting debug logs (client node as well as MDS server) during these operations? So that we can check out what the problem might be.

[Ben Evans (Inactive)]

drwxrwxrwx (I've tried many permutations around that theme, none work)

[Zhenyu Xu]

what's the permission/acl output of /mnt/blah ?

[Ben Evans (Inactive)]

ls -l /mnt displays everything correctly, but it is the only thing that does.

[Zhenyu Xu]

what's the output of "ls -l /mnt/"?

[Peter Jones]

Bobi

Could you please comment on this one?

Thanks

Peter