[LU-8799] User cannot access directories created by user

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Blocker
• Reporter: Cliff White (Inactive)
• Assignee: Oleg Drokin
• Resolution: Fixed
• Labels: soak

Environment

Soak, version=lustre: 2.8.59_79_gb8811a0

• Severity: 3

Description

All files on soak are created by userid 'soaktest'
Sequence of failures:
Mdtest started on 5 clients, job output:

mkdir: cannot create directory `/mnt/soaked/soaktest/test/mdtestfpp/670897': File exists
unable to open '/mnt/soaked/soaktest/test/mdtestfpp/670897': Permission denied (13)
error: setstripe: create file '/mnt/soaked/soaktest/test/mdtestfpp/670897' failed: Permission denied
error opening /mnt/soaked/soaktest/test/mdtestfpp/670897: Permission denied (13)
llapi_semantic_traverse: Failed to open '/mnt/soaked/soaktest/test/mdtestfpp/670897': Permission denied (13)
error: getstripe failed for /mnt/soaked/soaktest/test/mdtestfpp/670897.
error opening /mnt/soaked/soaktest/test/mdtestfpp/670897: Permission denied (13)
llapi_semantic_traverse: Failed to open '/mnt/soaked/soaktest/test/mdtestfpp/670897': Permission denied (13)
error: getdirstripe failed for /mnt/soaked/soaktest/test/mdtestfpp/670897.
error opening /mnt/soaked/soaktest/test/mdtestfpp/670897: Permission denied (13)
llapi_semantic_traverse: Failed to open '/mnt/soaked/soaktest/test/mdtestfpp/670897': Permission denied (13)
error: getdirstripe failed for /mnt/soaked/soaktest/test/mdtestfpp/670897.

attempting to do an 'ls' of the directory as user 'soak test'

[soaktest@lola-17 ~]$ ls -l /mnt/soaked/soaktest/test/mdtestfpp/
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670766: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670897: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670917: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670877: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670845: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670674: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670905: Permission denied
ls: cannot access /mnt/soaked/soaktest/test/mdtestfpp/670894: Permission denied
total 20
drwxr-xr-x 3 soaktest soaktest 8192 Nov  3 12:24 670670
d????????? ? ?        ?           ?            ? 670674
drwxr-xr-x 3 soaktest soaktest 4096 Nov  3 12:26 670729
drwxr-xr-x 3 soaktest soaktest 4096 Nov  3 12:26 670731
drwxr-xr-x 3 soaktest soaktest 4096 Nov  3 12:26 670749
d????????? ? ?        ?           ?            ? 670766
d????????? ? ?        ?           ?            ? 670845
d????????? ? ?        ?           ?            ? 670877
d????????? ? ?        ?           ?            ? 670894
d????????? ? ?        ?           ?            ? 670897
d????????? ? ?        ?           ?            ? 670905
d????????? ? ?        ?           ?            ? 670917

Comments

[Cliff White (Inactive)]

Client side errors:

lola-17.log:Nov  3 12:43:17 lola-17 kernel: LustreError: 56983:0:(llite_lib.c:2300:ll_prep_inode()) new_inode -fatal: rc -13
lola-17.log:Nov  3 12:43:17 lola-17 kernel: LustreError: 56983:0:(llite_lib.c:2300:ll_prep_inode()) Skipped 47 previous similar messages
lola-19.log:Nov  3 12:43:17 lola-19 kernel: LustreError: 25964:0:(llite_lib.c:2300:ll_prep_inode()) new_inode -fatal: rc -13
lola-19.log:Nov  3 12:43:17 lola-19 kernel: LustreError: 25964:0:(llite_lib.c:2300:ll_prep_inode()) Skipped 1 previous similar message
lola-25.log:Nov  3 12:43:19 lola-25 kernel: LustreError: 6420:0:(llite_lib.c:2300:ll_prep_inode()) new_inode -fatal: rc -13
lola-25.log:Nov  3 12:43:19 lola-25 kernel: LustreError: 6420:0:(llite_lib.c:2300:ll_prep_inode()) Skipped 3 previous similar messages

[Cliff White (Inactive)]

During the test period (short)
213 tests ran without errors, created directories/files correctly.
51 tests failed. with the above errors.

[Cliff White (Inactive)]

The error is also oddly transient.
ls -l as soak test from a single client will fail repeatedly with permission errors, then will work.

[Di Wang]

And also for some failed directories, once we change to "root", it will work again. And also it seems all of "-13" failures on MDS comes from permission check. I wonder if this is related with those patches from LU-3289? Could some security experts to have a look?

[Oleg Drokin]

The problem is a configuration issue.

[root@lola-11 ~]# id soaktest
id: soaktest: No such user
...
[root@lola-9 ~]# id soaktest
uid=539(soaktest) gid=539(soaktest) groups=539(soaktest)

So basically every time you try to acess this "unknownuser-owned" dir you get permission denied.

[Oleg Drokin]

there's the telltale stuff in /var/log/secure too (hinted by John to check there):

Nov  4 11:20:11 lola-11 l_getidentity[9316]: no such user 539
Nov  4 11:20:11 lola-11 l_getidentity[9316]: partial write ret -1: Invalid argument

[Cliff White (Inactive)]

Apparently when lola-11 was re-imaged, the users were not set up correctly.
Have filed DCO-6238, will address.

[Cliff White (Inactive)]

YP was not setup when lola-11 was re-installed, fixed

[Cliff White (Inactive)]

System issue - YPbind not setup