https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5546 Lustre <p>Aug 26 07:55:01 tscc-login1 postfix/qmgr<span class="error">&#91;3091&#93;</span>: 80B853C070: removed<br/> Aug 26 10:33:10 tscc-login1 kernel: LustreError: 11295:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:10 tscc-login1 kernel: LustreError: 11475:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:10 tscc-login1 kernel: LustreError: 11475:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 239 previous similar messages<br/> Aug 26 10:33:11 tscc-login1 kernel: LustreError: 11784:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:11 tscc-login1 kernel: LustreError: 11784:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 408 previous similar messages<br/> Aug 26 10:33:13 tscc-login1 kernel: LustreError: 12494:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:13 tscc-login1 kernel: LustreError: 12494:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 942 previous similar messages<br/> Aug 26 10:33:17 tscc-login1 kernel: LustreError: 13818:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:17 tscc-login1 kernel: LustreError: 13818:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 1743 previous similar messages<br/> Aug 26 10:33:25 tscc-login1 kernel: LustreError: 16658:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:25 tscc-login1 kernel: LustreError: 16658:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 3763 previous similar messages<br/> Aug 26 10:33:41 tscc-login1 kernel: LustreError: 22267:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:33:41 tscc-login1 kernel: LustreError: 22267:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 7418 previous similar messages<br/> Aug 26 10:34:13 tscc-login1 kernel: LustreError: 34084:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:34:13 tscc-login1 kernel: LustreError: 34084:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 15550 previous similar messages<br/> Aug 26 10:35:17 tscc-login1 kernel: LustreError: 57966:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:35:17 tscc-login1 kernel: LustreError: 57966:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 31707 previous similar messages<br/> Aug 26 10:37:25 tscc-login1 kernel: LustreError: 47314:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:37:25 tscc-login1 kernel: LustreError: 47314:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 69915 previous similar messages<br/> Aug 26 10:41:41 tscc-login1 kernel: LustreError: 20908:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:41:41 tscc-login1 kernel: LustreError: 20908:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 140545 previous similar messages<br/> Aug 26 10:50:13 tscc-login1 kernel: LustreError: 4158:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 10:50:13 tscc-login1 kernel: LustreError: 4158:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 346337 previous similar messages<br/> Aug 26 11:00:13 tscc-login1 kernel: LustreError: 50703:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 11:00:13 tscc-login1 kernel: LustreError: 50703:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 347381 previous similar messages<br/> Aug 26 11:10:13 tscc-login1 kernel: LustreError: 50839:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 11:10:13 tscc-login1 kernel: LustreError: 50839:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 386631 previous similar messages<br/> Aug 26 11:20:13 tscc-login1 kernel: LustreError: 57406:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 11:20:13 tscc-login1 kernel: LustreError: 57406:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 397281 previous similar messages<br/> Aug 26 11:30:13 tscc-login1 kernel: LustreError: 64118:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 11:30:13 tscc-login1 kernel: LustreError: 64118:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 395513 previous similar messages<br/> ...</p> <p>Aug 26 11:50:13 tscc-login1 kernel: LustreError: 18441:0:(file.c:3077:ll_inode_revalidate_fini()) dolphin: revalidate FID <span class="error">&#91;0x200005416:0x15ef:0x0&#93;</span> error: rc = -116<br/> Aug 26 11:50:13 tscc-login1 kernel: LustreError: 18441:0:(file.c:3077:ll_inode_revalidate_fini()) Skipped 402712 previous similar messages</p> Lustre server 2.4.2<br/> Linux dolphin-mds-9-2.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux LU-5546

Client getting "revalidate FID" errors, a lot of them

Bug Major Resolved Won't Fix Emoly Liu Haisong Cai sdsc Tue, 26 Aug 2014 20:26:57 +0000 Sat, 24 Mar 2018 14:02:58 +0000 Sat, 24 Mar 2018 14:02:58 +0000 Lustre 2.5.2 0 5 <p>Emoly</p> <p>Does this seem related to <a href="https://jira.whamcloud.com/browse/LU-4084" title="ll_inode_revalidate_fini()) failure -13" class="issue-link" data-issue-key="LU-4084"><del>LU-4084</del></a>/4522? If not, what do you suggest here?</p> <p>Thanks</p> <p>Peter</p> <p>I will look into this one.</p> <p>Hi Cai,<br/> Did this happen on a NFS export? because -116 means stale NFS handle.<br/> Are there any error messages on MDS? If possible, could you please upload MDS logs? Thanks.</p> <p>This message shouldn't be printed onto the console at all for -ESTALE (-116), since this is a common error that can be hit during normal operation. There is some underlying problem that is causing the revalidate to be called repeatedly (400k calls in 20s by the end of the log) that also needs to be fixed.</p> <p>Cai, is there a client that is hung during this time? The question of NFS exports is also important. Also, it would be great if you could get a stack trace for when this is happening (ideally through systemtap, but also possibly via repeatedly trying sysrq-p) to see what the callpath is. It isn't clear why the caller isn't handling -ESTALE correctly and actually moving on to some new FID.</p> <p>Hi Andreas,</p> <p>The error showed after we rebooted MDS (because of <a href="https://jira.whamcloud.com/browse/LU-5503" title="MDS (2.4.2) are getting &quot;Service thread ... inactive&quot; and file-system times out" class="issue-link" data-issue-key="LU-5503"><del>LU-5503</del></a>) which took about 1hour.</p> <p>We have since reboot that client as well and it is now error free.<br/> I will remember to collect more info next when I see it.</p> <p>thanks,<br/> Haisong</p> <p>SDSC have moved onto more current releases so I do not think any further work is needed here</p> Development End date Mon, 1 Dec 2014 20:26:57 +0000 Rank 1|hzwuon: Rank (Obsolete) 15465 Severity Start date Tue, 26 Aug 2014 20:26:57 +0000