[LU-2518] Corrupted files

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.8
• Type: Bug
• Priority: Minor
• Reporter: Chris Locke (Inactive)
• Assignee: Bruno Faccini (Inactive)
• Resolution: Fixed
• Labels: mn8

Environment

RHEL 6.2

• Severity: 3

Description

End customer is Lunds University, who has support for our hardware and lustre through us. I have done some basic troubleshooting with them, had them try rm -f, and unlink, but they cannot perform any operations on the files. I suggested the run a file system check, but they cannot take the filesystem down for that. They realize they may have hit a bug, and upgrading could fix the issue, but they would really like to find a way to remove the files right now without going through that yet. I'll attach the logs I have, and also below you can find the last email I have from the customer, which provides a good summary for the issue.

-----From Customer-----

The problematic files were sockets/pipes defined on a server not using
Lustre, and rsynced into Lustre from that server. That went just fine.
The next step we did was copying the files from one part of our Lustre
fs to another, and thereby acquiring proper ACLs. This has worked fine
for all normal files, directories and links, but these sockets have
turned into something broken, that we can't remove.

A little googling brought this up:

http://jira.whamcloud.com/browse/LU-784

As we're on 1.8.8, it seems very similar to our problem. What I'd like
to know, is if there's someone from WhamCloud (or Intel, these days...)
that can give us any hints on what we can do, other than upgrading to
Lustre 2.X? I'm guessing there are ways to clear the faulty inodes
directly from the MDS and/or OSTs, but I'd need some guidance for that.
We'd really like to have this fixed before talking about a Lustre upgrade...

Issue Links

• is duplicated by: LU-784 Cannot access named pipes or sockets ...

Comments

[Chris Locke (Inactive)]

I should add, I am a Support Engineer for NetApp

[Peter Jones]

Bruno will help with this issue

[Bruno Faccini (Inactive)]

I am setting the necessary stuff to reproduce the issue on our test system and will try to get you a procedure to fix/remove the files asap.

[Bruno Faccini (Inactive)]

Seems that mounting the MDT as ldiskfs on the MDS and then issue "setfacl -b <mount-point>/ROOT/<relative-path-to-file>" fixes the problem "live" without any issue on still mounting-clients that even recently accessed the affected files/pipes/sockets.

Looks like the problem comes from the fact setting ACL on such files in 1.8 does half the work ... and thus triggers errors upon further access.

[Bruno Faccini (Inactive)]

Since "setfacl -b" resets all ACLs, you may also want to unset specific/non-default ACLs on affected files, still via/under the ldiskfs-mount of MDT, by first displaying them with "getfacl" and then selectivelly remove them with "setfacl -x".

Also, having alook to the concerned source code and ACL/EA content with debugfs seems that setfacl is doing things well but further actions (stat()) on file are missing/expecting something more (EA.LOV ?) to correctly interpret it ...

[Zhenyu Xu]

here is the fix patch (should fix MDS code) http://review.whamcloud.com/4887

commit message

LU-2518 mds: handle reply buffer correctly

* obd_valid is a 64-bit value, mds_shrink_reply()'s 3rd and 4th param
  takes boolean value, we need make proper conversion.
* Fix glitch in mds_shrink_reply().
* Add test cases.

[Peter Jones]

Thanks Bobijam. Will the same fix apply to b1_8? The customer is running 1.8.8-wc1.

[Zhenyu Xu]

yes, this patch is only for b1_8. b2_x uses different ptlrpc pack method (new).

[Peter Jones]

Hmm. But the patch seems to be based on b2_1?!

[Zhenyu Xu]

sorry, my fault, pushed to the wrong branch, fix it now.

[Bruno Faccini (Inactive)]

Hello Chris,

Did you read all the updates ??

Do you and/or your customer feel comfortable with the rescue procedure I described ("mount -t ldiskfs"/"setfacl" on MDS) to fix corrupted files "live" ?? If not, feel free to ask for more details from me.

Also, Zhenyu has already submitted a fix to handle the root bug and it is currently under testing, and should be included in next 1.8 release.

Last, are named-pipes/sockets of a common use by your customer ?? Do you think new could be created to run applications in-place ?? Or were they only old stuff left that was just carried by the rsync ?? Are you and the customer aware that rsync has options to enable/disable these kind of files xfer ??

[Chris Locke (Inactive)]

Bruno,

Yes, I kept up over the weekend, and forwarded the info to my customer. I doubt I'll hear anything back till Wednesday due to the holiday, but as soon as I do I'll let you know the response. Thank you all for working on this.

[Chris Locke (Inactive)]

Customer let me know that they are uncomfortable applying the patch themselves, and would like to know if Whamcloud could possibly do a remote session with them to go through it. If you guys are willing to do this I could get you in contact with the customer.

[Chris Locke (Inactive)]

Got another response back, here it is:

> Last, are named-pipes/sockets of a common use by your customer ??

Nope, not on the Lustre FS (so far).

> Do you think new could be created to run applications in-place ??

Probably not, but as we're not sure what SW-packages we'll have to install for our bioinformatics people in the future, we can't rule it out.

> Or were they only old stuff left that was just carried by the rsync ??

In this case, yes. A "full" rsync copy from another server was done to Lustre, and then when parts of was to be moved into another directory on Lustre, these were accidentally included.

> Are you and the customer aware that rsync has options to enable/disable these kind of files xfer ??

We do now... Mostly we've been after the "copy everything, keep all info" approach when using rsync, so we tend to use rsync -a.

As we have a planned and promised upgrade of Lustre this spring (date
not decided yet) to a stable 2.X, and can't foresee any immediate need
for more named-pipes/socket in our Lustre FS, we're not sure if we need
to patch our setup of 1.8, but we'd like to know:

1) Are these "untouchable" files in any way harmful to the FS? Could
there be inode allocation problems, or anything like that? On a system
level, we guess that we could just avoid them during backup and hence
more or less forget about them until our upgrade.

2) The process described to unset the ACLs on the specific files isn't
clear enough for me to just jump in and fix it. Could it be specified in
a better step-by-step version? Should I mount the MDT on the active or
the non-active MDS? Is mounting the FS on an MDS a standard procedure
for debugging/fixing? I'm not sure if our contract allows us to ask for
hands-on (remote console) help in solving a task such as this, but it
would be nice to know.

As you might have guessed by now, we're a bit cautious about breaking
the FS right now - it's been a long and repetitive process setting it
up, with backup problems as an added bonus.

3) I stated earlier that we're on version 1.8.8 of Lustre, which turns
out to be only partly true - our clients are, but the MDS & ODS-machines
are still on 1.8.7. If we decide that patching is needed after all,
would the patch apply to 1.8.7? Should it be applied to all Lustre
machines (MDS, ODS, clients...) or a specific set?

Cheers,

/Mattias

[Bruno Faccini (Inactive)]

Hello Chris,

Your answers are what I expected, so we can easily presume that no more/new named pipes/sockets will be created.

About customer's questions, here are my answers :

1) the currently affected files will not cause any other problem than the known error each time they are accessed. In fact, the Inode is not corrupted it is just mis-interpreted by the MDS layer.

2) the work-around I found to fix the problem is not what we can say a "standard" procedure for fixing Inodes on the MDS side, and particulary if you run it live/in-parallel of the file-system being mounted and used. So if you/users can leave with such files, just wait for filesystem scheduled down-time to apply it. This said, the exact procedure/commands can be described as :

_ on the running/primary MDS mount the MDT device with ldiskfs, "mkdir </mount-point> ; mount -t ldiskfs <MDT-device> </mount-point>".

_ then on each of the affected named-pipe/socket, run the "setfacl -b <mount-point>/ROOT/<relative-path-to-file>" command (where <relative-path-to-file> is the relative path to the file starting at the file-system root from Lustre/Clients point of view) reset all its ACLs to the default set. If you don't know the exact list/paths of the files having the problem, you may be able to find potential ones by running a "find <LustreFS-mountpoint> -type p -print" and "find <LustreFS-mountpoint> -type s -print" from any Client, and then try to access their content via any command like "file" and see if you get "ERROR: cannot open <path/file> (Operation not supported)" error.

3) as I wrote for 1) the issue is on the MDS side so I presume (Zhenyu correct me if I am wrong, my understanding is that only the MDS side is wrong and has to be fixed here ...) an upgrade of the MDS/Servers should be ok for this particular problem but as usual partial upgrade has to be done only under very specific circumstances and can not be a scenario fully tested nor validated from our side.

Is this enough detailled and clear for you to report to the customer ??
Best regards and don't hesitate to ask more/again.
Bruno.

[Mattias Borell (Inactive)]

Hi Bruno!

This is the customer speaking...

1) OK, good to know.

2) The work-around seems clearer now. I'll have to discuss it with my colleagues as to if/when we can try this. We had a backup-crisis recently (had to classify all used tapes as rubbish, due to mechanical problems), and are slowly resyncing 140-150TB to tape. We'll probably wait at least until that's finished...

3) OK, so you'd recommend that all machines (MDS, ODS, clients) are patched/updated to the same level? If so, we'll probably wait until we can go for 2.X sometime during spring 2013. Is step 2) necessary to do before an upgrade to 2.X?

Cheers,

/Mattias, Lund University

[Bruno Faccini (Inactive)]

Hello Mattias,

To be complete on item 3), let say that even if Zhenyu (who developed the patch) confirms the issue is on the Server/MDS side only and that no regression should exist between Servers running next 1.8 version including the patch and Clients left with 1.8.7, this will be an un-tested environment so there is still the possibility of an hidden issue.

Prior to join WhamCloud/Intel team I was working for Bull and mainly at a customer site and each time it was decided to go this (unbalanced) way, because their Lustre datas are mission-critical, we have been running our own regression test-suite during hours prior to run with this configuration for production workload.

But frankly I don't think that the problem you currently face here can justify the work+risk we discuss.

Hope this helps and clarifies.
Best Regards.
Bruno.

[Bruno Faccini (Inactive)]

BTW, I forgot to confirm that build #11616 from patch Set #5 do not show the problem anymore, this in full Clients+Servers 1.8.8 configuration.

Any news/update from Site/NetApp side ??

[Mattias Borell (Inactive)]

Hi!

We're still not done with our backups, but as soon as we're satisfied that we have it all on tape (I'm tempted to add more tapedrives... ) we'll try the live fix. Should happen later this week, me thinks. I'll let you know how it works out.

/Mattias, Lund University

[Mattias Borell (Inactive)]

Hi!

OK, we've finally been able to execute the "live fix", and it seems to have worked just fine!

Thanks a bundle - your response was quick, tested and accurate, couldn't really ask for more.

/Mattias, now back at planning for expansion & upgrade of our Lustre setup... :->

[Bruno Faccini (Inactive)]

Mattias, Chris,
Do you think we can close this ticket now ?
Thank's again and in advance for your help and answers.
Best Regards.
Bruno.

[Mattias Borell (Inactive)]

Maybe I should have stated that more clearly in my last comment...

I'm fine with closing this ticket - our problem has been solved!

Thanks,

/Mattias

[Bruno Faccini (Inactive)]

Ok thank's, so closing as resolved !!