https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1451 Lustre <p>When I try to list the lustre file system I get the following error:</p> <p><span class="error">&#91;root@client02 ~&#93;</span># ls -l /scratch<br/> ls: cannot access /scratch/weisshorn: Permission denied<br/> total 0<br/> d????????? ? ? ? ? ? weisshorn</p> <p>Into the /var/log/messages I found these error:</p> <p>May 30 17:09:21 weisshorn01 kernel: LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21267: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 344us<br/> May 30 17:09:21 weisshorn01 kernel: LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 13366 previous similar messages<br/> ...<br/> ...<br/> May 30 17:19:32 mds01 kernel: LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21021: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 312us<br/> May 30 17:19:32 mds01 kernel: LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 2305 previous similar messages</p> <p>-----------<br/> dmesg</p> <p>Lustre: MDS scratch-MDT0000 disabled MDS fid capability.<br/> Lustre: MDS scratch-MDT0000 disabled OSS fid capability.<br/> LustreError: 12856:0:(mdt_lproc.c:451:lprocfs_wr_identity_info()) scratch-MDT0000: invalid data count = 2, size = 1048<br/> Lustre: 12853:0:(upcall_cache.c:420:upcall_cache_flush_one()) scratch-MDT0000: flush entry ffff880e22bcb940: key 0, ref 0, fl 0, cur 1338387613, ex 5594562238/5595732233<br/> LustreError: 12838:0:(genops.c:1422:obd_export_evict_by_uuid()) scratch-MDT0000: can't disconnect 0: no exports found<br/> Lustre: 12859:0:(mdt_lproc.c:415:lprocfs_wr_identity_upcall()) scratch-MDT0000: identity upcall set to 0<br/> LustreError: 12867:0:(mdt_lproc.c:971:lprocfs_mdt_wr_mdc()) scratch-MDT0000: no export 0 found<br/> LNet: 12921:0:(nidstrings.c:1061:parse_nidrange()) can't parse nidrange: "0"<br/> Lustre: 12921:0:(mdt_lproc.c:869:lprocfs_wr_nosquash_nids()) scratch-MDT0000: failed to set nosquash_nids to "0<br/> ", can't parse: rc -22<br/> Lustre: scratch-MDT0000: root_squash is set to 0:0<br/> LustreError: 25565:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 374us<br/> LustreError: 25644:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 239us<br/> LustreError: 25617:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 269us<br/> LustreError: 25617:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 10 previous similar messages<br/> LustreError: 18008:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 251us<br/> LustreError: 18008:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 38 previous similar messages<br/> LustreError: 7033:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21658: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 311us<br/> LustreError: 7033:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 50 previous similar messages<br/> LustreError: 25599:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21858: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 290us<br/> LustreError: 25599:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 15 previous similar messages<br/> Lustre: MDS scratch-MDT0000 disabled MDS fid capability.<br/> Lustre: MDS scratch-MDT0000 disabled OSS fid capability.<br/> LustreError: 13918:0:(genops.c:1422:obd_export_evict_by_uuid()) scratch-MDT0000: can't disconnect 0: no exports found<br/> Lustre: 13939:0:(mdt_lproc.c:415:lprocfs_wr_identity_upcall()) scratch-MDT0000: identity upcall set to 0<br/> LustreError: 13948:0:(mdt_lproc.c:971:lprocfs_mdt_wr_mdc()) scratch-MDT0000: no export 0 found<br/> LustreError: 13935:0:(mdt_lproc.c:451:lprocfs_wr_identity_info()) scratch-MDT0000: invalid data count = 2, size = 1048<br/> LNet: 13989:0:(nidstrings.c:1061:parse_nidrange()) can't parse nidrange: "0"<br/> Lustre: 13989:0:(mdt_lproc.c:869:lprocfs_wr_nosquash_nids()) scratch-MDT0000: failed to set nosquash_nids to "0<br/> ", can't parse: rc -22<br/> Lustre: scratch-MDT0000: root_squash is set to 0:0<br/> LustreError: 9554:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 315us<br/> LustreError: 9554:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 17 previous similar messages<br/> LustreError: 20244:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 241us<br/> LustreError: 20244:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 184 previous similar messages<br/> LustreError: 18065:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 258us<br/> LustreError: 18065:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 434 previous similar messages<br/> LustreError: 6171:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 254us<br/> LustreError: 6171:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 1099 previous similar messages<br/> LustreError: 7139:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 229us<br/> LustreError: 7139:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 4693 previous similar messages<br/> LustreError: 6148:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21078: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 230us<br/> LustreError: 6148:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 12704 previous similar messages<br/> LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21522: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 310us<br/> LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 19373 previous similar messages<br/> LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21579: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 400us<br/> LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 8452 previous similar messages<br/> LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21936: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 298us<br/> LustreError: 25630:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 11701 previous similar messages<br/> LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21267: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 344us<br/> LustreError: 20450:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 13366 previous similar messages<br/> LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) scratch-MDT0000: error invoking upcall 0 scratch-MDT0000 21021: rc -2; check /proc/fs/lustre/mdt/scratch-MDT0000/identity_upcall, time 312us<br/> LustreError: 7032:0:(mdt_identity.c:145:mdt_identity_do_upcall()) Skipped 2305 previous similar messages<br/> ...</p> <p>Thanks<br/> Regards</p> <p>Fabio</p> Linux weisshorn02.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 #1 SMP Tue Apr 24 14:27:35 PDT 2012 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-1451

Permission Denied when listing lustre file system

Task Critical Resolved Fixed Cliff White Fabio Verzelloni Wed, 30 May 2012 11:37:10 +0000 Fri, 1 Jun 2012 11:46:58 +0000 Fri, 1 Jun 2012 11:46:58 +0000 Lustre 2.2.0 0 1 <p>The user has to be in the MDS passwd files, or you must set the upcall so that the MDS can lookup user permissions.</p> <p>I had to reboot the entire cluster to recover from the situation, after that the file system was back in a better shape, but now I'm experiencing a weird behavior about drop of performance and I/O stuck ( <a href="http://jira.whamcloud.com/browse/LU-1455" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1455</a> ).</p> <p>Situation seems to be fine, the only question is, is it normal that into the log file there are few entries about:</p> <p>May 31 15:00:57 weisshorn03 kernel: LustreError: 6092:0:(ldlm_lib.c:2697:target_bulk_io()) Skipped 23 previous similar messages<br/> May 31 16:01:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;d062c2&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 17:01:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;200854&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 18:10:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;b127f8&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 19:10:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;16231b&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 20:10:31 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;16e9e8&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 21:10:46 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;90cde7&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 22:11:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;ef438d&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> May 31 23:11:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;0e0f76&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 00:20:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;52255a&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 01:20:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;9cf92e&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 02:20:31 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;ed7263&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 03:07:08 weisshorn03 rhsmd: This system is registered to RHN Classic<br/> Jun 1 03:20:46 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;dcc233&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 04:21:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;efd79f&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 05:21:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;a7c4c9&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 06:30:01 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;68079a&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 07:30:16 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;6afb66&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 08:30:31 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;e45d32&#93;</span> ldap_result() failed: Can't contact LDAP server<br/> Jun 1 09:30:46 weisshorn03 nslcd<span class="error">&#91;4016&#93;</span>: <span class="error">&#91;9b500d&#93;</span> ldap_result() failed: Can't contact LDAP server</p> <p>Which is the best practice to configure the LDAP on the Lustre servers?</p> <p>Fabio</p> <p>LDAP is not a part of Lustre, we have no recommendations for a best setup. Whatever works for your site. Yes, the error messages are not good. <br/> However, since you have solved the issue with permissions, I am closing this bug. Please open up a new ticket if you have new issues. </p> <p>customer rebooted. please reopen if necessary.</p> Development Rank 1|hzw03z: Rank (Obsolete) 10097