[LU-1451] Permission Denied when listing lustre file system

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.2.0
• Type: Task
• Priority: Critical
• Reporter: Fabio Verzelloni
• Assignee: Cliff White (Inactive)
• Resolution: Fixed

Environment

Linux weisshorn02.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 #1 SMP Tue Apr 24 14:27:35 PDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Description

When I try to list the lustre file system I get the following error:

[root@client02 ~]# ls -l /scratch
ls: cannot access /scratch/weisshorn: Permission denied
total 0
d????????? ? ? ? ? ? weisshorn

Into the /var/log/messages I found these error:

May 30 17:09:21 weisshorn01 kernel: LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21267: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 344us
May 30 17:09:21 weisshorn01 kernel: LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 13366 previous similar messages
...
...
May 30 17:19:32 mds01 kernel: LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21021: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 312us
May 30 17:19:32 mds01 kernel: LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 2305 previous similar messages

-----------
dmesg

Lustre: MDS scratch-MDT0000 disabled MDS fid capability.
Lustre: MDS scratch-MDT0000 disabled OSS fid capability.
LustreError: 12856:0:(mdt_lproc.c:451:lprocfs_wr_identity_info()) scratch-MDT0000: invalid data count = 2, size = 1048
Lustre: 12853:0:(upcall_cache.c:420:upcall_cache_flush_one()) scratch-MDT0000: flush entry ffff880e22bcb940: key 0, ref 0, fl 0, cur 1338387613, ex 5594562238/5595732233
LustreError: 12838:0:(genops.c:1422:obd_export_evict_by_uuid()) scratch-MDT0000: can't disconnect 0: no exports found
Lustre: 12859:0:(mdt_lproc.c:415:lprocfs_wr_identity_upcall()) scratch-MDT0000: identity upcall set to 0
LustreError: 12867:0:(mdt_lproc.c:971:lprocfs_mdt_wr_mdc()) scratch-MDT0000: no export 0 found
LNet: 12921:0:(nidstrings.c:1061:parse_nidrange()) can't parse nidrange: "0"
Lustre: 12921:0:(mdt_lproc.c:869:lprocfs_wr_nosquash_nids()) scratch-MDT0000: failed to set nosquash_nids to "0
", can't parse: rc -22
Lustre: scratch-MDT0000: root_squash is set to 0:0
LustreError: 25565:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 374us
LustreError: 25644:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 239us
LustreError: 25617:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 269us
LustreError: 25617:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 10 previous similar messages
LustreError: 18008:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 251us
LustreError: 18008:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 38 previous similar messages
LustreError: 7033:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 311us
LustreError: 7033:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 50 previous similar messages
LustreError: 25599:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21858: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 290us
LustreError: 25599:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 15 previous similar messages
Lustre: MDS scratch-MDT0000 disabled MDS fid capability.
Lustre: MDS scratch-MDT0000 disabled OSS fid capability.
LustreError: 13918:0:(genops.c:1422:obd_export_evict_by_uuid()) scratch-MDT0000: can't disconnect 0: no exports found
Lustre: 13939:0:(mdt_lproc.c:415:lprocfs_wr_identity_upcall()) scratch-MDT0000: identity upcall set to 0
LustreError: 13948:0:(mdt_lproc.c:971:lprocfs_mdt_wr_mdc()) scratch-MDT0000: no export 0 found
LustreError: 13935:0:(mdt_lproc.c:451:lprocfs_wr_identity_info()) scratch-MDT0000: invalid data count = 2, size = 1048
LNet: 13989:0:(nidstrings.c:1061:parse_nidrange()) can't parse nidrange: "0"
Lustre: 13989:0:(mdt_lproc.c:869:lprocfs_wr_nosquash_nids()) scratch-MDT0000: failed to set nosquash_nids to "0
", can't parse: rc -22
Lustre: scratch-MDT0000: root_squash is set to 0:0
LustreError: 9554:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 315us
LustreError: 9554:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 17 previous similar messages
LustreError: 20244:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 241us
LustreError: 20244:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 184 previous similar messages
LustreError: 18065:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 258us
LustreError: 18065:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 434 previous similar messages
LustreError: 6171:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 254us
LustreError: 6171:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 1099 previous similar messages
LustreError: 7139:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 229us
LustreError: 7139:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 4693 previous similar messages
LustreError: 6148:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 230us
LustreError: 6148:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 12704 previous similar messages
LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21522: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 310us
LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 19373 previous similar messages
LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21579: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 400us
LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 8452 previous similar messages
LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21936: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 298us
LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 11701 previous similar messages
LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21267: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 344us
LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 13366 previous similar messages
LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21021: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 312us
LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 2305 previous similar messages
...

Thanks
Regards

Fabio

Comments

[Cliff White (Inactive)]

The user has to be in the MDS passwd files, or you must set the upcall so that the MDS can lookup user permissions.

[Fabio Verzelloni]

I had to reboot the entire cluster to recover from the situation, after that the file system was back in a better shape, but now I'm experiencing a weird behavior about drop of performance and I/O stuck ( http://jira.whamcloud.com/browse/LU-1455 ).

[Fabio Verzelloni]

Situation seems to be fine, the only question is, is it normal that into the log file there are few entries about:

May 31 15:00:57 weisshorn03 kernel: LustreError: 6092:0:(ldlm_lib.c:2697:target_bulk_io()) Skipped 23 previous similar messages
May 31 16:01:01 weisshorn03 nslcd[4016]: [d062c2] ldap_result() failed: Can't contact LDAP server
May 31 17:01:16 weisshorn03 nslcd[4016]: [200854] ldap_result() failed: Can't contact LDAP server
May 31 18:10:01 weisshorn03 nslcd[4016]: [b127f8] ldap_result() failed: Can't contact LDAP server
May 31 19:10:16 weisshorn03 nslcd[4016]: [16231b] ldap_result() failed: Can't contact LDAP server
May 31 20:10:31 weisshorn03 nslcd[4016]: [16e9e8] ldap_result() failed: Can't contact LDAP server
May 31 21:10:46 weisshorn03 nslcd[4016]: [90cde7] ldap_result() failed: Can't contact LDAP server
May 31 22:11:01 weisshorn03 nslcd[4016]: [ef438d] ldap_result() failed: Can't contact LDAP server
May 31 23:11:16 weisshorn03 nslcd[4016]: [0e0f76] ldap_result() failed: Can't contact LDAP server
Jun 1 00:20:01 weisshorn03 nslcd[4016]: [52255a] ldap_result() failed: Can't contact LDAP server
Jun 1 01:20:16 weisshorn03 nslcd[4016]: [9cf92e] ldap_result() failed: Can't contact LDAP server
Jun 1 02:20:31 weisshorn03 nslcd[4016]: [ed7263] ldap_result() failed: Can't contact LDAP server
Jun 1 03:07:08 weisshorn03 rhsmd: This system is registered to RHN Classic
Jun 1 03:20:46 weisshorn03 nslcd[4016]: [dcc233] ldap_result() failed: Can't contact LDAP server
Jun 1 04:21:01 weisshorn03 nslcd[4016]: [efd79f] ldap_result() failed: Can't contact LDAP server
Jun 1 05:21:16 weisshorn03 nslcd[4016]: [a7c4c9] ldap_result() failed: Can't contact LDAP server
Jun 1 06:30:01 weisshorn03 nslcd[4016]: [68079a] ldap_result() failed: Can't contact LDAP server
Jun 1 07:30:16 weisshorn03 nslcd[4016]: [6afb66] ldap_result() failed: Can't contact LDAP server
Jun 1 08:30:31 weisshorn03 nslcd[4016]: [e45d32] ldap_result() failed: Can't contact LDAP server
Jun 1 09:30:46 weisshorn03 nslcd[4016]: [9b500d] ldap_result() failed: Can't contact LDAP server

Which is the best practice to configure the LDAP on the Lustre servers?

Fabio

[Cliff White (Inactive)]

LDAP is not a part of Lustre, we have no recommendations for a best setup. Whatever works for your site. Yes, the error messages are not good.
However, since you have solved the issue with permissions, I am closing this bug. Please open up a new ticket if you have new issues.

[Cliff White (Inactive)]

customer rebooted. please reopen if necessary.