https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4084 Lustre <p>Our Lustre client reported the following errors continuously:</p> <p>Oct 10 16:06:13 avmmst1a kernel: Lustre: Mounted modelfs-client<br/> Oct 10 16:06:13 avmmst1a kernel: Lustre: Mounted preposfs-client<br/> Oct 10 16:06:18 avmmst1a kernel: LustreError: 54029:0:(mdc_locks.c:736:mdc_enqueue()) ldlm_cli_enqueue: -13<br/> Oct 10 16:06:18 avmmst1a kernel: LustreError: 54029:0:(file.c:2196:ll_inode_revalidate_fini()) failure -13 inode 452984833<br/> Oct 10 16:06:25 avmmst1a kernel: LustreError: 54029:0:(mdc_locks.c:736:mdc_enqueue()) ldlm_cli_enqueue: -13<br/> Oct 10 16:06:25 avmmst1a kernel: LustreError: 54029:0:(file.c:2196:ll_inode_revalidate_fini()) failure -13 inode 159383553</p> <p>We've googled but no hint. What's the cause of this? Thanks.</p> <p>Regards,<br/> Patrick</p> Lustre 2.1.5 on CentOS 6.3 LU-4084

ll_inode_revalidate_fini()) failure -13

Story Major Resolved Not a Bug Lai Siyao Patrick Chan Thu, 10 Oct 2013 08:15:25 +0000 Sat, 16 Nov 2013 14:26:40 +0000 Sat, 16 Nov 2013 14:26:40 +0000 0 5 <p>Could you check messages on MDS, is there similar error reported?</p> <p>Could you also check if Clients and Servers share the same UIDs/GIDs databases ??</p> <p>Lai Siyao,</p> <p>There is no logs in MDS.</p> <p>Bruno,</p> <p>MDS is also a NIS server, both servers and clients use the same NIS server.</p> <p>Patrick</p> <p>The next time you see this error, could you use `lctl dk` on MDS to dump debug logs right after that?</p> <p>I run 'lctl dk' on MDS, the file lustre_debug.log.gz was uploaded to ftp.whamcloud.com/uploads.</p> <p>By the way, I try to find out if the culprit is the inode 159383553. I perform a full scan on the inode 159383553, but can't find any file in the whole lustre file system.</p> <p>client&gt; find /lustre_mnt_point -inum 159383553</p> <p>I don't find any error messages in this log, which means -13 (-EACCES) is not from disk filesystem, because mdt_getattr_internal() will print an error if it gets attr from disk fails. Did you dump the debug logs right after you see this failure? Because debug log size is limited, and it only contains the most recent logs. </p> <p>To further understand this failure, could you enable more debug on MDS with `lctl set_param debug=+inode` and `lctl set_param debug=+trace` which will enable debug for inode access and function trace. And you can use `lctl get_param debug_mb` and `lctl set_param debug_mb=&lt;debug_size&gt;` to check and increase debug memory size.</p> <p>You can also dump debug logs on client, which may help you find the file name.</p> <p>The 'll_inode_revalidate_fini())' failure message appears occasionally, I need to watch /var/log/messages on client and capture the debug buffer on MDS promptly.</p> <p>The log file lustre_debug2.log.gz is uploaded. This is the debug buffer generated about 10 seconds after 'll_inode_revalidate_fini()) failure' appears on client.</p> <p>As you suggested, I've add inode and trace into debug parameter, debug buffer size is increased to 512MB.</p> <p>Just already discovered that both inodes mentioned in the error (159383553 &amp; 452984833) are owned by top level directory of mount point.</p> <p>It'll be better to do this in a script if system is busy, in case the old logs get discarded.</p> <p>We've solved the problem.</p> <p>The error is displayed, when Nagios periodically runs the plugin check_disk on lustre client to check the disk capacity of the Lustre file system.</p> <p>However, the user nagios only exist on master node (lustre client). MDS does not have such user.</p> <p>After adding local user nagios on MDS, this error does not appear anymore.</p> <p>That's great! Thanks for letting us know.</p> Development Rank 1|hzw5b3: Rank (Obsolete) 10976