[LU-7842] ACL's applied over NFS are not consistent when looping file operations

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Dave Bond (Inactive)

Description

We are experiencing an issue where on a lustre client we do not see the issue, but exported over NFS we see this issue on one of our three production lustre file systems. We cannot reproduce this issue on any other system, but is causing us production issues on our oldest lustre instance.

Running the script attached over NFS after a few iterations we hit the following issue:

[joe59240@vws250 joe59240]$ /dls/tmp/joe59240/stresstest
....mkdir: cannot create directory `5': Permission denied

Each "." is an iteration of the loop as you will see in the script

This persists for maybe as long as five seconds before files can be written to the folder again and the script runs.

At the moment we have not ever had the script run to completion, but on other lustre file systems we can run it hundreds of times to completion.
The file system has many different NFS exporters each exporting a different folder in the root of the file system. As is common practice on all other systems at Diamond. We can re produce this on all exporters attached to this particular file system.

We are thinking after a number of weeks looking at the issue that it is not the exporter as this is across all servers that export lustre but down to the interaction between FS and NFS.

We have put in a few sleeps in the script to try and identify if there is a buffering issue where we are modifying or deleting before a flush to disk. But this has not improved the symptoms.

Would it be possible to advise further debugging?

Issue Links

• is related to: LU-7630 permission denied over NFS

Comments

[Peter Jones]

ok so either this is no longer happening or you are no longer concerned about it. Either way, I'll close out the ticket

[Peter Jones]

Frederik

Any news?

Peter

[Frederik Ferner (Inactive)]

Peter,

the patch last week was a bit late for that maintenance window, so we had to wait until this week.

We have applied the patch on the MDS this morning and so far we've not been able to reproduce the issue, though if I remember right, this sometimes had been the case immediately after rebooting the NFS server. And we did have to reboot the NFS server as it suffered a LBUG after finishing recovery. We're looking into this and if we can't find anything in Jira, we'll open another ticket for that.

Thanks,
Frederik

[Peter Jones]

Dave/Frederik

Have you applied the supplied diagnostic patch?

Peter

[Lai Siyao]

the patch for b2_7_fe is on: http://review.whamcloud.com/#/c/20992/

[Lai Siyao]

okay, I'll do it now.

[Peter Jones]

Lai

While LU-8305 may prevent this change completing testing on master, it should have no relevance to Diamond running on the 2.7 FE branch so could you please port the patch there for them to try?

Thanks

Peter

[Lai Siyao]

The autotest failure looks to be caused by LU-8305. I'll watch the progress of that ticket.

[Frederik Ferner (Inactive)]

Lai,

I've looked at the patch, it has received a '-1' from maloo but I can't work out if this is a failure that is seen elsewhere. I think it might be but would like to double check before considering to apply this patch for a test. Considering we are unfortunately only seeing this on a production file system, and MDS changes require a full file system outage, we will need to schedule this and I currently can't promise when this will happen. Hopefully early next week if everything else looks good.

Thanks,
Frederik

[Lai Siyao]

hi Dave, I just pushed a patch, which changes MDS code only, could you apply it and test again?