[LU-1152] ls: cannot open directory .: Permission denied

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 1.8.7
• Environment: Client: 1.8.7-wc1, 2.6.32-131.12.1.el6.x68_64; Server: 1.8.4
• Severity: 3

Epic

client

• Assignee: Zhenyu Xu
• Reporter: Ben Evans (Inactive)

Description

mount lustre on /mnt
ls /mnt (works)
cd /mnt/blah (works)
ls
ls: cannot open directory .: Permission denied
ls foo (works)
ls /mnt/blah (fails)
ls /mnt/blah/foo (works)

Running strace on ls, I get permission denied opening '.', regardless of directory permissions.

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA

Comments

[Ben Evans (Inactive)]

server kernel version is 2.6.18-194.3.1.0.1.el5 and Lustre 1.8.4
Formatted with that kernel/Lustre version.
8 OSTs, default striping.

Steps on the client are exactly as you describe, mount, mkdir, ls.

Disabling selinux on the client fixed the ls issue.

[Zhenyu Xu]

Ben,

Can you describe more info about the step? It's 1.8.4 Lustre server, and all server devices were formated with this 1.8.4 Lustre server? What Linux kernel version do these servers use? What steps did you take? Just mounted the Lustre system and "mkdir /mnt/testdir" and then "ls /mnt/testdir" failed?

[Xuezhao Liu]

First, the selinux should be disabled.

Secondly, for "fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61" such log, in the case that only 1 OST installed on the server, is it possible the EA(stripe info) be empty? Seems it is just a kind of warning/log message, not really error.

[Ben Evans (Inactive)]

disabling selinux is a workaround, but not a fix.

Since 1.8.7-wc1 clients work on older versions of Linux with the same filesystem, this looks like a kernel incompatibility, rather than a client/server issue (or one that Lustre aggravates).

[Zhenyu Xu]

I think not, gotta use >= 1.8.7 server if your client is 1.8.7.

Since old MDS does not have the essential info (stripe info for client) as following MDS log shows.

$ grep fsfilt_ldiskfs_get_md /tmp/logs/1152/dump.mds.out
00002000:00000040:21:1330696880.700393:0:12708:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:9:1330696880.706343:0:12706:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112901473: rc -61
00002000:00000040:9:1330696880.707520:0:12710:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114144769: rc -61
00002000:00000040:15:1330696880.712199:0:12714:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112899589: rc -61
00002000:00000040:17:1330696880.713367:0:12723:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1114275841: rc -61
00002000:00000040:19:1330696880.714589:0:31534:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112965121: rc -61
00002000:00000040:15:1330696880.715786:0:31557:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866819: rc -61
00002000:00000040:17:1330696880.716954:0:31555:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113817091: rc -61
00002000:00000040:1:1330696880.718103:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113882625: rc -61
00002000:00000040:23:1330696880.720901:0:12707:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866822: rc -61
00002000:00000040:22:1330696880.722087:0:31556:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1113849857: rc -61
00002000:00000040:4:1330696939.175531:0:31531:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:17:1330696943.714265:0:31561:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 1112866817: rc -61
00002000:00000040:15:1330696943.715212:0:12720:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:7:1330696952.152840:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444164: rc -61
00002000:00000040:7:1330696952.152854:0:12719:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330696957.721309:0:12711:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:3:1330696981.345390:0:12713:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:13:1330696991.321460:0:31535:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61
00002000:00000040:17:1330697014.615160:0:31536:0:(fsfilt-ldiskfs.c:717:fsfilt_ldiskfs_get_md()) error getting EA 4/lov from inode 708444163: rc -61

[Peter Piela (Inactive)]

Would disabling selinux be a workaround for this problem?

[Zhenyu Xu]

client request testdir's xattr

00000080:00200000:2:1330700770.421348:0:16553:0:(xattr.c:349:ll_getxattr()) VFS Op:inode=708444163/3303540295(ffff88010e00ed50), xattr security.selinux
...
00000002:00000001:2:1330700770.421799:0:16553:0:(mdc_request.c:367:mdc_xattr_common()) Process leaving via err_out (rc=18446744073709551555 : -61 : ffffffffffffffc3)
====>ENODATA

new client(1.8.7) old server(1.8.4) syndrome, I am afraid you'd update your server.

[Ben Evans (Inactive)]

Process for the dump:

mount client
start debugging
ls /mnt
ls -l /mnt
mkdir /mnt/testdir
ls -l /mnt/testdir (failed)
touch /mnt/testdir/testfile
ls -l /mnt/testdir (failed)
ls -l /mnt/testdir/testfile (success)
rm /mnt/testdir/testfile
rmdir /mnt/testdir (failed)
chmod 777 /mnt/testdir
rmdir /mnt/testdir (failed)

[Zhenyu Xu]

Would you mind enabling all debug message level "lctl set_param debug=-1" on client and servers, and collecting debug logs (client node as well as MDS server) during these operations? So that we can check out what the problem might be.

[Ben Evans (Inactive)]

drwxrwxrwx (I've tried many permutations around that theme, none work)

[Zhenyu Xu]

what's the permission/acl output of /mnt/blah ?