[LU-7093] mkdir: cannot create directory: Operation not permitted

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Critical
• Reporter: Giuseppe Di Natale (Inactive)
• Assignee: Di Wang
• Resolution: Fixed
• Labels: dne, llnl, zfs

Environment

lustre-2.7.57-2.6.32_504.16.2.1chaos.ch5.3.x86_64_gea38322.x86_64
Redhat 2.6.32-504.16.2.1chaos.ch5.3.x86_64

• Severity: 3

Description

In a a set of directories on ldne, /p/ldne/dinatale/count4_index*, Joe is able to create a file, but not a directory. Joe owns the directories and the mode is set to 700.

In other directories, for example /p/ldne/dinatale/count3_index*, Joe is able to create both files and directories successfully. They have the same ownership and mode.

zwicky96@dinatale:pwd
/p/ldne/dinatale/count4_index0

zwicky96@dinatale:ls -al
total 54
drwx------  2 dinatale lcstaff 43008 Sep  1 14:13 .
dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 ..

zwicky96@dinatale:date > something

zwicky96@dinatale:ls -l something
-rw------- 1 dinatale dinatale 29 Sep  1 15:45 something

zwicky96@dinatale:rm something

zwicky96@dinatale:ls -al                                                                          
total 54
drwx------  2 dinatale lcstaff 43008 Sep  1 15:46 .
dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 ..

zwicky96@dinatale:mkdir something
mkdir: cannot create directory `something': Operation not permitted

zwicky96@dinatale:ls -al            
total 54
drwx------  2 dinatale lcstaff 43008 Sep  1 15:46 .
dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 ..

zwicky96@dinatale:rpm -q lustre
lustre-2.7.57-2.6.32_504.16.2.1chaos.ch5.3.x86_64_gea38322.x86_64

zwicky96@dinatale:distro_version
toss 2.3-6.ch5.3

zwicky96@dinatale:lfs getdirstripe .
.
lmv_stripe_count: 4 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
     0           [0x200000453:0x1ff:0x0]
     1           [0x280000411:0x1ff:0x0]
     2           [0x24000041c:0x1fe:0x0]
     3           [0x2c000040b:0x1fd:0x0]

zwicky96@dinatale:pwd
/p/ldne/dinatale/count4_index0

zwicky96@dinatale:ls -ld ../count4_index*
drwx------ 2 dinatale lcstaff 43008 Sep  1 15:46 ../count4_index0
drwx------ 2 dinatale lcstaff 43008 Sep  1 14:00 ../count4_index1
drwx------ 2 dinatale lcstaff 43008 Sep  1 14:00 ../count4_index2
drwx------ 2 dinatale lcstaff 43008 Sep  1 14:00 ../count4_index3

zwicky96@dinatale:ls -ld ../count3_index*
drwx------ 3 dinatale lcstaff 32256 Sep  1 14:06 ../count3_index0
drwx------ 3 dinatale lcstaff 32256 Sep  1 14:06 ../count3_index1
drwx------ 3 dinatale lcstaff 32256 Sep  1 14:06 ../count3_index2
drwx------ 3 dinatale lcstaff 32256 Sep  1 14:06 ../count3_index3

We are running 4 MDSs each w/ 1 MDT, 2 OSSs each w/ 1 OST, and clients are running the same lustre version as the servers. Backend is zfs.

Issue Links

• is related to: LU-7013 replay-single test_110c: FAIL: create...
• is related to: LU-6341 Do not check security when accessing ...

Comments

[Giuseppe Di Natale (Inactive)]

Linked LU-7013. That issue may or may not be related, but it is a test involving a mkdir.

[Olaf Faaland]

Attached debug log output from the MDS nodes, dumped after "mkdir something" which failed as Joe described.

See call to mdt_md_create() on line 29 of dne-count4.debug.zwicky-ldne-mds1.after

We can gather other information as required - just let us know.

[Olaf Faaland]

Some FIDs in case they help with interpreting logs:

/p/ldne/dinatale/count4_index0
lmv_stripe_count: 4 lmv_stripe_offset: 0
mdtidx FID[seq:oid:ver]
0 [0x200000453:0x1ff:0x0]
1 [0x280000411:0x1ff:0x0]
2 [0x24000041c:0x1fe:0x0]
3 [0x2c000040b:0x1fd:0x0]

/p/ldne: [0x200000007:0x1:0x0]
/p/ldne/dinatale: [0x200000404:0x1:0x0]
/p/ldne/dinatale/count4_index0: [0x200000457:0x1208:0x0]

zwicky96@dinatale:sudo lfs getdirstripe /p/ldne/dinatale
/p/ldne/dinatale
lmv_stripe_count: 0 lmv_stripe_offset: 0

[Di Wang]

Hmm, did you set default directory stripeEA on count4_index0 ? Please check.

lfs getdirstripe -D count4_index0

According to the debug log, it seems it try to create a remote directory under the count4_index0

00000004:00000002:9.0:1441148647.363955:0:21353:0:(mdt_reint.c:308:mdt_md_create()) @@@ Create  (something->[0x200000479:0x583a:0x0]) in [0x2c000040b:0x1fd:0x0]  req@ffff880ea0a31cc0 x1510173626909920/t0(0) o36->548c6f4a-55be-90f6-da7e-6f3b85da262e@192.168.120.96@o2ib6:178/0 lens 616/568 e 0 to 0 dl 1441148708 ref 1 fl Interpret:/0/0 rc 0/0

Something seems on MDT0, but its name entry being assigned to MDT3. If that is what you want, then you need do this on all of your MDS

[root@testnode tests]# ../utils/lctl set_param mdt.*.enable_remote_dir_gid=-1

This will allow all of non-sysadmin users to create cross-MDT objects (striped, remote dir etc) on all of MDTs.

btw: for normal user to test striped directory, you better include the fix http://review.whamcloud.com/13990, which seems not in 2.7.57 yet. So either upgrade to 2.7.58, or apply the fix in your build. thanks.

[Giuseppe Di Natale (Inactive)]

Here is the output from lfs regarding default stripe for count4_index0.

zwicky99@dinatale:lfs getdirstripe -D count4_index0
(Default)count4_index0
lmv_stripe_count: 4 lmv_stripe_offset: 0

Here is the same lfs output for count3_index0 for comparison.

zwicky99@dinatale:lfs getdirstripe -D count3_index0 
(Default)count3_index0
lmv_stripe_count: 3 lmv_stripe_offset: 0

Again, mkdir fails in count4_index0.

zwicky99@dinatale:pwd
/p/ldne/dinatale/count4_index0
zwicky99@dinatale:mkdir something
mkdir: cannot create directory `something': Operation not permitted

But, in count3_index0 it is successful.

zwicky99@dinatale:pwd
/p/ldne/dinatale/count3_index0
zwicky99@dinatale:mkdir something
zwicky99@dinatale:ll
total 12
drwx------ 2 dinatale dinatale 11776 Sep  3 10:05 something

enable_remote_dir_gid was never set on any of the MDTs and directory creation seems to work in other directories. User dinatale is a non-sysadmin user already. The build we are using to do our testing also appears to include the fix you linked according to our git log.

[Di Wang]

The reason it did not fail on count3_index0 is that its name and directory itself are happen to be assigned both MDT0, so it is not a remote directory, then this remote_dir_gid check will not operate on this operation. But for count4_index0, the name and directory are being assigned to different MDT0, and then this remote_dir_gid check needs to step in.

Note for striped directory, when creating dir or files under it,

• their name entries will be assigned according its name hash value (FNV-1a hash algorithm by default), stripe_index = hash_value % stripe_count;
• If there are not default stripeEA, then the directory itself will be assigned to the same MDT as its name entry, otherwise it will be assigned to the MDT where default stripeEA indicate, which means creating the directory (mkdir something) under a striped directory, which also have default stripeEA (like your case), might create a remote directory.

So without enable_remote_dir_gid , you will also meet this problem under count3_index0, try to create some directories with other name, like mkdir something1.

[Giuseppe Di Natale (Inactive)]

Di, thank you for the explanation. It makes sense now.

I still find it odd that a user may be allowed to make a striped directory even though they are not allowed to actually create remote directories. Wouldn't it be better behavior to not permit users to make striped directories if remote directories is disabled? Likewise, if remote directories are enabled and a striped directory exists in the filesystem, then remote directories cannot be disabled?

[Di Wang]

I still find it odd that a user may be allowed to make a striped directory even though they are not allowed to actually create remote directories. 

hmm, this maybe a problem. I will work on a fix, thanks for finding this.

[Andreas Dilger]

I think the check needs to be that if a striped directory exists, the users can create subdirectories under the striped directory regardless of the extra /proc setting. Also, setting the default directory striping itself needs to check the extra /proc setting, but once it is set then the regular user can create subdirectories according to this policy.

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16286
Subject: LU-7093 mdt: Remote operation permission check
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 482a09045f2d8be0681146c01c7c7a248ed0f559

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16286/
Subject: LU-7093 mdt: Remote operation permission check
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 703195acc2157969e66cb07946522b918dfd7dea

[Joseph Gmitter (Inactive)]

Landed for 2.8.