https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9015 Lustre <p>We have a curious problem where a user is managing to wedge a few COS6 compute nodes every day to the point they don't even respond to the console. The only thing we see logged before this happens is:</p> <p>Jan 12 15:59:14 n602 kernel: <span class="error">&#91;3463048.109323&#93;</span> LustreError: 21705:0:(file.c:3256:ll_inode_revalidate_fini()) fouo5: revalidate FID <span class="error">&#91;0x2000068c4:0x16fd0:0x0&#93;</span> error: rc = -4</p> <p>I have no idea if this is a symptom of the node breaking down in some way unrelated to Lustre or if this is part of the cause, so I'd like to figure out what this error means. After a quick look at the code this should be the return code from either md_intent_lock or md_getattr, but I can't find where the error code -4 is defined.</p> <p>Any tips?</p> lustre ee client-2.5.42.4-2.6.32_642.6.2 LU-9015

Meaning of revalidate FID rc=-4

Question/Request Minor Resolved Fixed Yang Sheng Peter Bortas Thu, 12 Jan 2017 18:20:39 +0000 Thu, 19 Jan 2017 16:59:46 +0000 Thu, 19 Jan 2017 13:38:20 +0000 Lustre 2.5.0 0 3 <p>-4 is EINTR, and it likely has nothing to do with the hang at hand. This particular one was silenced in <a href="https://jira.whamcloud.com/browse/LU-6627" title="Client inode close failed: ll_close_inode_openhandle())" class="issue-link" data-issue-key="LU-6627"><del>LU-6627</del></a> I believe.<br/> It means that an application got some sort of a signal (like somebody pressed ^C or some other sources) in the middle of waiting for RPC reply from MDS.<br/> application pid was 21705 if you somehow can track what it was back then.<br/> It's defined in linux kernel source in include/uapi/asm-generic/errno-base.h file.</p> <p>Now if you want to investigate the hang, you probably want to do something like sysrq-t, do you have other debugging stuff enabled? nmi watchdogs, serial console attached? Are you setup to collect crashdumps?</p> <p>Yang Sheng</p> <p>Could you please assist with this one as further informaiton is supplied</p> <p>Thanks</p> <p>Peter</p> <p>Thanks Oleg,</p> <p>That makes sense. It looks like (probably) all occurrences of this hang corresponds with the users jobs getting preemted and thus killed by Slurm.</p> <p>We do not currently have sysrq-t enabled in the node images, but I'll enable that and roll out new images during the week. Serial console is available via IPMI but we are not set up to collect crash dumps. Making a device available for crash dumps might not be possible with the limited space available.</p> <p>We pushed out sysrq-enabled images Yesterday, and unfortunately this hang does not look like it can be escaped with sysrq on the console or break via SOL.</p> <p>Lustre is now only one of many suspects since the error message probably has nothing to do with it. So I'm fine with closing this ticket for now. I can reopen it if further testing on our side points more fingers at Lustre.</p> <p>Many thanks, Peter. </p> <p>Just a late note in case you did not know, but you don't need a dedicated crashdump device for every node.<br/> You can have an nfs share where all nodes dump to once the need arises, this is kind of a setup that is very common and I do it too.</p> <p>I highly recommend you to look into setting something like that up (it does not even need to be crazy fast or have redundant storage, just a cheap huge multiterabate HDD in a node that's up all the time and has nfs server on it.)</p> <p>Having this setup early would save you tons of trouble later when you actually need a crashdump from something to debug some other problem.</p> <p>I did not know you could crash-dump over network filesystems. Very interesting, thanks!</p> <p>Dump capability is now added to the roadmap for our cluster environment development.</p> Development Rank 1|hzz0if: Rank (Obsolete) 9223372036854775807