[LU-4084] ll_inode_revalidate_fini()) failure -13

• Type: Story
• Resolution: Not a Bug
• Priority: Major
• Environment: Lustre 2.1.5 on CentOS 6.3
• Assignee: Lai Siyao
• Reporter: Patrick Chan (Inactive)

Description

Our Lustre client reported the following errors continuously:

Oct 10 16:06:13 avmmst1a kernel: Lustre: Mounted modelfs-client
Oct 10 16:06:13 avmmst1a kernel: Lustre: Mounted preposfs-client
Oct 10 16:06:18 avmmst1a kernel: LustreError: 54029:0:(mdc_locks.c:736:mdc_enqueue()) ldlm_cli_enqueue: -13
Oct 10 16:06:18 avmmst1a kernel: LustreError: 54029:0:(file.c:2196:ll_inode_revalidate_fini()) failure -13 inode 452984833
Oct 10 16:06:25 avmmst1a kernel: LustreError: 54029:0:(mdc_locks.c:736:mdc_enqueue()) ldlm_cli_enqueue: -13
Oct 10 16:06:25 avmmst1a kernel: LustreError: 54029:0:(file.c:2196:ll_inode_revalidate_fini()) failure -13 inode 159383553

We've googled but no hint. What's the cause of this? Thanks.

Regards,
Patrick

Comments

[Peter Jones]

That's great! Thanks for letting us know.

[Patrick Chan (Inactive)]

We've solved the problem.

The error is displayed, when Nagios periodically runs the plugin check_disk on lustre client to check the disk capacity of the Lustre file system.

However, the user nagios only exist on master node (lustre client). MDS does not have such user.

After adding local user nagios on MDS, this error does not appear anymore.

[Lai Siyao]

It'll be better to do this in a script if system is busy, in case the old logs get discarded.

[Patrick Chan (Inactive)]

Just already discovered that both inodes mentioned in the error (159383553 & 452984833) are owned by top level directory of mount point.

[Patrick Chan (Inactive)]

The 'll_inode_revalidate_fini())' failure message appears occasionally, I need to watch /var/log/messages on client and capture the debug buffer on MDS promptly.

The log file lustre_debug2.log.gz is uploaded. This is the debug buffer generated about 10 seconds after 'll_inode_revalidate_fini()) failure' appears on client.

As you suggested, I've add inode and trace into debug parameter, debug buffer size is increased to 512MB.

[Lai Siyao]

I don't find any error messages in this log, which means -13 (-EACCES) is not from disk filesystem, because mdt_getattr_internal() will print an error if it gets attr from disk fails. Did you dump the debug logs right after you see this failure? Because debug log size is limited, and it only contains the most recent logs.

To further understand this failure, could you enable more debug on MDS with `lctl set_param debug=+inode` and `lctl set_param debug=+trace` which will enable debug for inode access and function trace. And you can use `lctl get_param debug_mb` and `lctl set_param debug_mb=<debug_size>` to check and increase debug memory size.

You can also dump debug logs on client, which may help you find the file name.

[Patrick Chan (Inactive)]

I run 'lctl dk' on MDS, the file lustre_debug.log.gz was uploaded to ftp.whamcloud.com/uploads.

By the way, I try to find out if the culprit is the inode 159383553. I perform a full scan on the inode 159383553, but can't find any file in the whole lustre file system.

client> find /lustre_mnt_point -inum 159383553

[Lai Siyao]

The next time you see this error, could you use `lctl dk` on MDS to dump debug logs right after that?

[Patrick Chan (Inactive)]

Lai Siyao,

There is no logs in MDS.

Bruno,

MDS is also a NIS server, both servers and clients use the same NIS server.

Patrick

[Bruno Faccini (Inactive)]

Could you also check if Clients and Servers share the same UIDs/GIDs databases ??