[LU-7842] ACL's applied over NFS are not consistent when looping file operations

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Dave Bond (Inactive)

Description

We are experiencing an issue where on a lustre client we do not see the issue, but exported over NFS we see this issue on one of our three production lustre file systems. We cannot reproduce this issue on any other system, but is causing us production issues on our oldest lustre instance.

Running the script attached over NFS after a few iterations we hit the following issue:

[joe59240@vws250 joe59240]$ /dls/tmp/joe59240/stresstest
....mkdir: cannot create directory `5': Permission denied

Each "." is an iteration of the loop as you will see in the script

This persists for maybe as long as five seconds before files can be written to the folder again and the script runs.

At the moment we have not ever had the script run to completion, but on other lustre file systems we can run it hundreds of times to completion.
The file system has many different NFS exporters each exporting a different folder in the root of the file system. As is common practice on all other systems at Diamond. We can re produce this on all exporters attached to this particular file system.

We are thinking after a number of weeks looking at the issue that it is not the exporter as this is across all servers that export lustre but down to the interaction between FS and NFS.

We have put in a few sleeps in the script to try and identify if there is a buffering issue where we are modifying or deleting before a flush to disk. But this has not improved the symptoms.

Would it be possible to advise further debugging?

Issue Links

• is related to: LU-7630 permission denied over NFS

Comments

[Lai Siyao]

the patch for b2_7_fe is on: http://review.whamcloud.com/#/c/20992/

[Lai Siyao]

okay, I'll do it now.

[Peter Jones]

Lai

While LU-8305 may prevent this change completing testing on master, it should have no relevance to Diamond running on the 2.7 FE branch so could you please port the patch there for them to try?

Thanks

Peter

[Lai Siyao]

The autotest failure looks to be caused by LU-8305. I'll watch the progress of that ticket.

[Frederik Ferner (Inactive)]

Lai,

I've looked at the patch, it has received a '-1' from maloo but I can't work out if this is a failure that is seen elsewhere. I think it might be but would like to double check before considering to apply this patch for a test. Considering we are unfortunately only seeing this on a production file system, and MDS changes require a full file system outage, we will need to schedule this and I currently can't promise when this will happen. Hopefully early next week if everything else looks good.

Thanks,
Frederik

[Lai Siyao]

hi Dave, I just pushed a patch, which changes MDS code only, could you apply it and test again?

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20920
Subject: LU-7842 nfs: don't drop cap for getattr too
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8f76d6583ebb377200f14c4d547dca73d6581d1b

[Lai Siyao]

I don't find any clue in debug logs, so the -13 might be generated from NFS code (though it may be caused by lustre code, but it may be wrong attribute fetched to NFS server from MDS).

I'll see whether I can make a patch to add some debug messages.

[Dave Bond (Inactive)]

From the time stamp it has been 6 days since the last update when I uploaded the latest logs. Any chance of an update even to say you are still looking? We have shared this ticket number with the developers whom this is causing pain and I would like to provide them with an update.

[Dave Bond (Inactive)]

Just uploaded new dump file.

This was collected by

sudo lctl debug_daemon start /tmp/lustre-dump-14-05-16
and
stopped after the error had shown up on the NFS client.

[Lai Siyao]

I do see -13 error code in nfs logs, but I'm afraid the lustre debug log is not dumped in time, and the related logs were discarded (lustre only keeps a certain amount of debug logs in memory).

Could you modify your test script a bit to check errors of each command, and dump logs just upon error?