[LU-3727] LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body->valid & OBD_MD_FLID) failed Created: 08/Aug/13  Updated: 14/Jun/18  Resolved: 10/Feb/15

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.1.5, Lustre 1.8.9, Lustre 2.4.1
Fix Version/s: Lustre 2.7.0

Type: Bug Priority: Minor
Reporter: Oz Rentas Assignee: Emoly Liu
Resolution: Fixed Votes: 0
Labels: patch

Attachments: Text File log.txt     File log.unlink08.lctl.dk.out.gz     File lustre.log     File unlink08.c    
Issue Links:
Related
is related to LU-5730 intermittent I/O errors for some dire... Resolved
Severity: 3
Rank (Obsolete): 9597

 Description   

At GE Global Research, we ran into an LBUG with a 1.8.9 client that is re-exporting 2.1.5 Lustre:

Jul 31 10:26:46 scinfra3 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Jul 31 10:26:46 scinfra3 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Jul 31 10:26:46 scinfra3 kernel: NFSD: starting 90-second grace period
Jul 31 10:26:53 scinfra3 ntpd[8318]: synchronized to 3.40.208.30, stratum 2
Jul 31 10:29:46 scinfra3 kernel: LustreError: 27396:0:(llite_nfs.c:281:ll_get_parent()) ASSERTION(body->valid & OBD_MD_FLID) failed
Jul 31 10:29:46 scinfra3 kernel: LustreError: 27396:0:(llite_nfs.c:281:ll_get_parent()) LBUG
Jul 31 10:29:46 scinfra3 kernel: Pid: 27396, comm: nfsd
Jul 31 10:29:46 scinfra3 kernel:
Jul 31 10g:29:46 scinfra3 kernel: Call Trace:
Jul 31 10:29:46 scinfra3 kernel: [ ] libcfs_debug_dumpstack+0x51/0x60 [libcfs]
Jul 31 10:29:46 scinfra3 kernel: [ ] lbug_with_loc+0x7a/0xd0 [libcfs]
Jul 31 10:29:46 scinfra3 kernel: [ ] tracefile_init+0x0/0x110 [libcfs]
Jul 31 10:29:46 scinfra3 kernel: [ ] ll_get_parent+0x1e3/0x2b0 [lustre]
Jul 31 10:29:46 scinfra3 kernel: [ ] ll_get_dentry+0x6b/0xe0 [lustre]
Jul 31 10:29:46 scinfra3 kernel: [ ] mutex_lock+0xd/0x1d
Jul 31 10:29:46 scinfra3 kernel: [ ] find_exported_dentry+0x241/0x486 [exportfs]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_acceptable+0x0/0xdc [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] autoremove_wake_function+0x0/0x2e
Jul 31 10:29:46 scinfra3 kernel: [ ] sunrpc_cache_lookup+0x4b/0x128 [sunrpc]
Jul 31 10:29:46 scinfra3 kernel: [ ] exp_get_by_name+0x5b/0x71 [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] exp_find_key+0x89/0x9c [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_acceptable+0x0/0xdc [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] ll_decode_fh+0x197/0x240 [lustre]
Jul 31 10:29:46 scinfra3 kernel: [ ] set_current_groups+0x116/0x164
Jul 31 10:29:46 scinfra3 kernel: [ ] fh_verify+0x29c/0x4cf [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd3_proc_getattr+0x8a/0xbe [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_dispatch+0xd8/0x1d6 [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] svc_process+0x3f8/0x6bf [sunrpc]
Jul 31 10:29:46 scinfra3 kernel: [ ] __down_read+0x12/0x92
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x1a5/0x2cb [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] child_rip+0xa/0x11
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb [nfsd]
Jul 31 10:29:46 scinfra3 kernel: [ ] child_rip+0x0/0x11
Jul 31 10:29:46 scinfra3 kernel:

It appears to be easily reproducible, we are going to try to get a core dump, but I was wondering if there was anything obvious from this trace or any other jira tickets I might have missed. Also is there any other information that might be useful?

Thanks.



 Comments   
Comment by Peter Jones [ 08/Aug/13 ]

Thanks for the report Kit!

Comment by Peter Jones [ 09/Aug/13 ]

Emoly

What do you suggest here?

Peter

Comment by Emoly Liu [ 12/Aug/13 ]

Kit, could you please show how to reproduce this LBUG in detail? And a core dump file will be helpful. Thanks!

Comment by Kit Westneat (Inactive) [ 12/Aug/13 ]

Hi Emoly,

The customer is currently unable to reproduce after setting up kdump, so we are in a holding pattern. I will ask what they were doing to reproduce before.

Thanks.

Comment by Kit Westneat (Inactive) [ 13/Aug/13 ]

Hi Emoly,

We were able to capture a crash dump from a different client node. Strangely, this client gets a null pointer dereference when trying to print the LBUG stack trace.

Here is the vmcore:
http://ddntsr.com/ftp/2013-08-13-SR23991-vmcore

This is the kernel debuginfo for it:
http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-279.2.1.el6.x86_64.rpm
http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-279.2.1.el6.x86_64.rpm

crash> bt
PID: 23553  TASK: ffff881019a32080  CPU: 10  COMMAND: "nfsd"
 #0 [ffff88100a241520] machine_kexec at ffffffff8103281b
 #1 [ffff88100a241580] crash_kexec at ffffffff810ba792
 #2 [ffff88100a241650] text_poke at ffffffff815016f0
 #3 [ffff88100a241680] no_context at ffffffff81043bab
 #4 [ffff88100a2416d0] __bad_area_nosemaphore at ffffffff81043e35
 #5 [ffff88100a241720] bad_area_nosemaphore at ffffffff81043f03
 #6 [ffff88100a241730] __do_page_fault at ffffffff81044661
 #7 [ffff88100a241850] debugfs_kprobe_init at ffffffff815036ce
 #8 [ffff88100a241880] do_debug at ffffffff81500a85
 #9 [ffff88100a2419d8] libcfs_debug_dumpstack at ffffffffa01d78f5 [libcfs]
#10 [ffff88100a2419f8] lbug_with_loc at ffffffffa01d7f25 [libcfs]
#11 [ffff88100a241a48] libcfs_assertion_failed at ffffffffa01e0696 [libcfs]
#12 [ffff88100a241a98] ll_get_parent at ffffffffa08836f8 [lustre]
#13 [ffff88100a241b38] reconnect_path at ffffffffa021e3b0 [exportfs]
#14 [ffff88100a241ba8] exportfs_decode_fh at ffffffffa021e7aa [exportfs]
#15 [ffff88100a241d18] fh_verify at ffffffffa064abea [nfsd]
#16 [ffff88100a241da8] nfsd3_proc_getattr at ffffffffa0655b6c [nfsd]
#17 [ffff88100a241dd8] nfsd_dispatch at ffffffffa064743e [nfsd]
#18 [ffff88100a241e18] svc_process_common at ffffffffa05fb5d4 [sunrpc]
#19 [ffff88100a241e98] svc_process at ffffffffa05fbc10 [sunrpc]
#20 [ffff88100a241eb8] nfsd at ffffffffa0647b62 [nfsd]
#21 [ffff88100a241ee8] kthread at ffffffff81091d66
#22 [ffff88100a241f48] kernel_thread at ffffffff8100c14a

The customer uses NFS for staging files onto the clusters. So it should just be a lot of copies, removes, that sort of thing. There shouldn't be any jobs, but it's possible there are some desktop applications that run against it. It happens pretty frequently, though they don't have a reproducer.

Let me know if there is any other data I can get.

Thanks.

Comment by Li Xi (Inactive) [ 14/Aug/13 ]

We hit the same problem. Here is how we reproduce it:
1. Create a directory under the Lustre directory which is exported, e.g. create /mnt/lustre/export/dir.
2. Change the directory's mode to 700 so that no one other than root has the right to access it.
3. Export the Lustre directry, i.e. /mnt/lustre/export. Please make sure 'no_root_squash' option is not enabled.
4. Mount the NFS client on some node.
5. On the NFS client, cd into the Lustre directory, i.e. directory of /mnt/lustre/export.
6. On the NFS server (and the Lustre client), unexport the NFS server, unmount Lustre client,
7. On the NFS client, run 'ls -l'. The operation will stuck.
8. On the NFS server, remount Lustre client, reexport NFS server (i.e. simulate a reboot).
9. Wait for a while, we will hit the LBUG (on master branch) or get EACCES (on Lustre-2.1.4)

After some inverstigation, we found that the cause is that the nfs daemon user does not have the permission to access ".." directory.

We will upload the fix patch soon.

Comment by Li Xi (Inactive) [ 14/Aug/13 ]

Here is the patch which tries to fix the problem.
http://review.whamcloud.com/7327

Comment by Emoly Liu [ 14/Aug/13 ]

Thanks for your patch!

Comment by Li Xi (Inactive) [ 16/Aug/13 ]

Here is a patch which tries to fix the problem in another way:
http://review.whamcloud.com/#/c/7349/

The last patch (http://review.whamcloud.com/7327) tries to fix the problem by skipping the permission check on MDT. This patch avoids permission denied by pretending the client is sending a RPC of root user. It is useful for us because this is a client-side-only patch, and it is difficult for us to get downtime for the customer.

I am not sure which patch is better. Maybe we should think more about their security problem?

Comment by nasf (Inactive) [ 18/Aug/13 ]

I am not sure whether I understand your case correctly or not. But consider the following case:

1) root user "mkdir /tmp/test"
2) root user "chmod 0700 /tmp/test"
3) root user "cd /tmp/test"
4) root user "su USER1"
5) USER1 "ls -l"

The result is "ls: cannot open directory .: Permission denied".

Back to the LU-3727, if the "/mnt/lustre/export" is only accessible for root user, then "ls -l" under it as non-root user should get "Permission denied" instead of bypassing permission check as the patch will do, right? If yes, we need to remove invalid "ASSERT()" and report "Permission denied" as expected.

Comment by Li Xi (Inactive) [ 18/Aug/13 ]

Sorry, maybe my former explaination was not accurate. Here is how to reproduce the problem.
1) NFS server "mkdir /mnt/lustre/export"
2) NFS server exports /mnt/lustre/export
3) NFS server "mkdir /mnt/lustre/export/dir"
4) NFS server "chmod 0700 /mnt/lustre/export/dir"
5) NFS client "cd /mnt/lustre/export/"
6) NFS server "service nfs stop"
7) NFS client "ls -l"
8) NFS server "umount /mnt/lustre"
9) NFS server mount /mnt/lustre again
10) NFS server "service nfs start"
11) NFS client hits LBUG after a few seconds

Please notice that at step 7), we are under "/mnt/lustre/export", and we have the permission for this operation.
If "no_root_squash" is disabled, the problem will show up no matter the user is root or not.

Comment by Shuichi Ihara (Inactive) [ 26/Aug/13 ]

Hi FanYong & Emoly

Would you please review Li's patch quickly?
And please give us advices/comments on this, please?

Thanks!

Comment by nasf (Inactive) [ 26/Aug/13 ]

I think the patch (http://review.whamcloud.com/#/c/7349/) is some hack, any will introduce security hole. There may be UID/GID mapping on MDT side, so even if you specify the special case as root user, it still possible be mapped to another user in the future. It is MDT to decide how to handle such case.

Comment by Li Xi (Inactive) [ 26/Aug/13 ]

Thank you very much, Fan Yong!

I guess the first patch (http://review.whamcloud.com/7327) is more acceptable, right?

Comment by nasf (Inactive) [ 26/Aug/13 ]

I think 7327 is better than 7349, although the former one still can be improved.

Comment by Alexey Lyashkov [ 01/Oct/13 ]

Li Xi,

I have some questions about you script to reproduce a bug.
1) if nfs client will able to do 'cd $dir' they should be lookup a permissions in last directory as minimum.
so you should be have EACCESS in that step.
2) in case nfs client don't have any verify permissions in that step - have a EACCESS in return for ls -l command, looks reasonable, but you tried avoid that permission check and take output.

if (2) is true - i think question - why 'md_getattr_name' isn't return an error as it's expected.

may you attach a full lustre debug from lustre in that crash? i think you use a single node configuration for a nfs server so we will be see all operations in logs.

Comment by Li Xi (Inactive) [ 01/Oct/13 ]

The trace output when this BUG happens. Please note that this log is trace on Lustre-2.1, so no LBUG happens. But basically, it is the same problem.

Comment by Li Xi (Inactive) [ 01/Oct/13 ]

Hi Alexey,

Yeah, NFS client has the permission to access '/mnt/lustre/export/' but it has no permission to access '/mnt/lustre/export/dir'. However, when NFS daemon restart (which is not normal and that is when ll_get_parent() is called), NFS client should get the attribute of '/mnt/lustre/export/dir/../', which will make Lustre client (and NFS server) hit the LBUG. Normally, NFS client could access '/mnt/lustre/export/' withouth any problem though it has no permission to access '/mnt/lustre/export/dir', but ll_get_parent() is different. Since there is no reason to require that a user doing 'ls -l /mnt/lustre/export/' has the permission to access '/mnt/lustre/export/dir', I think the best way to fix this is to avoid the permission check of '/mnt/lustre/export/dir' in this case.

I've post the lustre debug file as 'log.txt'. Sorry, I should have post it earlier.

Comment by Patrick Farrell (Inactive) [ 23/Oct/13 ]

We encountered this assertion while running a set of tests from the Linux Test Project over NFS. We did not do any unmount/remounting of NFS or Lustre as part of this, but are consistently able to hit the bug with this test. We hit it with both a SLES11SP1 Lustre client and a CentOS 6.4 Lustre client, in both cases to CentOS 6.4 servers running 2.4.1.

I'll attached the source file for this particular test (the LTP is a GPL'ed project).

The test set is called unlink08, and is a series of tests of the unlink system call.

We're planning to test the patch in http://review.whamcloud.com/#/c/7327/ today, I'll report back with results.

Comment by Patrick Farrell (Inactive) [ 23/Oct/13 ]

With 3727 applied, the 'unlink8' test no longer hits this bug.

Comment by Alexey Lyashkov [ 23/Oct/13 ]

[root@rhel6-64 WC-review]# gcc unlink08.c
unlink08.c:119:18: error: test.h: No such file or directory
unlink08.c:120:21: error: usctest.h: No such file or directory

Comment by Patrick Farrell (Inactive) [ 23/Oct/13 ]

Sorry, Alexey - I hadn't intended that to be buildable/useable by itself, I just posted it for reference.

It's part of LTP, which can be downloaded from here:
http://sourceforge.net/projects/ltp/files/latest/download

You'll find it in testcases/kernel/syscalls/unlink/ in the untarred package, but you'll have to figure out how to build and run that specific test.

Comment by Alexey Lyashkov [ 23/Oct/13 ]

I have checked on ~2 days ago with some older ltp code, and don't able to replicate that assertion.
but my test env hit an hung while accessing an exported dir.

after hung finished

[root@rhel6-64 ltp]# export TMPDIR=/mnt/lustre2/; /Users/shadow/work/lustre/work/ltp/testcases/kernel/syscalls/unlink/unlink08 

unlink08    1  TPASS  :  unlink(<unwritable directory>) returned 0
unlink08    2  TPASS  :  unlink(<unsearchable directory>) returned 0
unlink08    3  TPASS  :  unlink(<directory>) Failed, errno=21

where

  1. cat /etc/exports
    /mnt/lustre/export 192.168.0.0/16(rw,no_root_squash,fsid=100)

/without fsid i have issues with exporting directory while root lustre dir exported fine/

ps. same for last LTP from git.

Comment by Patrick Farrell (Inactive) [ 23/Oct/13 ]

Interesting. I confirmed and we were able to cause that assertion by running only unlink8, as well as running unlink8 as part of the larger test suite.
No idea why we're seeing it and you aren't. Regardless, as mentioned above, the patch fixes that issue for us.

By the way...
You may already know the details on the FSID issue, but if not, they're here:
https://jira.hpdd.intel.com/browse/LU-3550
Very briefly, Lustre 2.4 uses 64 bit inodes. NFS does not support 64 bit root inodes, because it explicitly casts inodes to 32 bit in FSID generation, and when those generated FSIDs are returned by the client and unparsed by the server, they cannot be matched back to the 64 bit root inode on Lustre. (In contrast, when you provide an FSID with -o fsid, it is used directly, so this issue is avoided. Why a generated FSID is parsed to an inode and looked up by the server - rather than used as an arbitrary ID for the export, like when an FSID is provided - is something of a mystery.)

Comment by Alexey Lyashkov [ 23/Oct/13 ]

question about fsid - is long story. lustre put native fid with parent fid in NFS file handle, FSID may be any number and it's now same as block device id (for compatibility with older servers in pair). but main question - why i able to export /mnt/lustre without fsid set, but /mnt/lustre/export need fsid.

as about assert - did you able to take a crashdump and extract lustre debug log from core file?
and provide a git hash of lustre sources.

Comment by Patrick Farrell (Inactive) [ 23/Oct/13 ]

re FSID:
I'm guessing, but I believe because /mnt/lustre has an inode # in your normal Linux file system, whereas /mnt/lustre/export exists only in Lustre.
It may matter if you mount Lustre before or after exporting /mnt/lustre - Possibly if you mount lustre at /mnt/lustre/ first and then do the exportfs, you would get the Lustre inode # and see the problem.
I recall having problems exporting the root of Lustre mounts as well as sub-directories of those mounts, but I was definitely mounting Lustre before doing the export.

re: The assertion.
We were running Cray's version of Lustre 2.4.1, so no Git hash. However, it's very close to the Intel 2.4.1 tag.

We could probably extract logs from the dump we have, but the logs only had the default debugging options, IE:
debug=ioctl neterror warning error emerg ha config console

If you're still interested, I could probably get those for you tomorrow.

Comment by Alexey Lyashkov [ 23/Oct/13 ]

it's don't enough ... i need full debug logs for both - mdt and client nodes.. primary for a client, but mdt also interested. I will try checkout to 2.4.1 and see a difference.

tried to replicate with v2_4_1 tag, but without success.

Comment by Alexey Lyashkov [ 23/Oct/13 ]

Li,

attached log don't have an information about rpc and isn't full - it's have just last step when we hit an error..

Comment by Li Xi (Inactive) [ 24/Oct/13 ]

I reproduced the problem with following steps.
1. Run all the servers/client on the same machine.
[root@server1 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 100798068 92258100 3419652 97% /
none 16424096 0 16424096 0% /dev/shm
/dev/sda1 521780 216388 278888 44% /boot
/dev/sda5 1043548 43396 947140 5% /mnt/mgs
/dev/sda6 7865980 456800 6885408 7% /mnt/mdt0
/dev/sda7 7865980 456396 6885812 7% /mnt/mdt1
/dev/sda8 33442176 562620 31201168 2% /mnt/ost0
/dev/sda9 80025512 464196 75545068 1% /mnt/ost1
192.168.3.100@tcp:/server1
113467688 1026816 106746236 1% /mnt/lustre
2. NFS server is on the same machine. Please note no_root_squash should NOt be enabled to reproduce this problem using root
[root@server1 ~]# cat /etc/exports
/mnt/lustre/nfs *(fsid=0,rw)
3. The directory should have no execute permission for normal users.
[root@server1 ~]# ll /mnt/lustre/nfs/
total 4
drwx------ 2 root root 4096 Oct 24 11:39 dir
4. Trace the path
[root@server1 ~]# echo trace > /proc/sys/lnet/debug
5. Clearup the messages
[root@server1 ~]# lctl debug_kernel /tmp/lustre.log
6. Start the NFS
[root@server1 ~]# service nfs restart
7. Start the NFS client
[root@server1 ~]# mount 192.168.3.100:/mnt/lustre/nfs /mnt/nfs
8. Go down to the NFS client directory
[root@server1 ~]# cd /mnt/nfs
9. Get the target dentry so that ll_get_parent will call for it.
[root@server1 nfs]# ls -l
10. Get the target dentry so that ll_get_parent will call for it.
11. Stop the NFS server to simulate a reboot
[root@server1 nfs]# service nfs stop
Shutting down NFS daemon: [ OK ]
Shutting down NFS mountd: [ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [ OK ]
[root@server1 nfs]# umount /mnt/lustre
12. Access the directory again, this command will stuck
[root@server1 nfs]# ls -l
13. Start the NFS server again on another terminal
[root@server1 ~]# mount -t lustre 192.168.3.100@tcp:/server1 /mnt/lustre
[root@server1 ~]# service nfs start
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
Starting NFS daemon: [ OK ]
14. After a few seconds the stucked command hit the LBUG.
[root@server1 nfs]# ls -l

Message from syslogd@server1 at Oct 24 12:08:53 ...
kernel:LustreError: 3348:0:(llite_nfs.c:311:ll_get_parent()) ASSERTION( body->valid & (0x00000001ULL) ) failed:

Message from syslogd@server1 at Oct 24 12:08:53 ...
kernel:LustreError: 3348:0:(llite_nfs.c:311:ll_get_parent()) LBUG
15. Get the Lustre log
lctl debug_file /tmp/lustre-log.1382641733.3348 /tmp/lustre.log

Comment by Li Xi (Inactive) [ 24/Oct/13 ]

Alexey,

'lustre.log' is the trace log which I got when reproducing the problem. Hope it helps.

Comment by Alexey Lyashkov [ 24/Oct/13 ]

Li,

thanks!

Comment by Alexey Lyashkov [ 24/Oct/13 ]

Li,

thanks again. devil in details.. we need additional directory created in exported dir.
without it ll isn't trigger a bug.

Comment by Alexey Lyashkov [ 24/Oct/13 ]

as i talk before - mdt generate an error as before

00000004:00000001:1.0:1382635559.670116:0:15672:0:(mdd_permission.c:309:__mdd_permission_internal()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3)
00000004:00000001:1.0:1382635559.670117:0:15672:0:(mdd_dir.c:90:__mdd_lookup()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3)
00000004:00000001:1.0:1382635559.670117:0:15672:0:(mdd_dir.c:115:mdd_lookup()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3)

but that error isn't returned to the caller

00000004:00000001:1.0:1382635559.670119:0:15672:0:(mdt_handler.c:1273:mdt_getattr_name_lock()) Process leaving (rc=0 : 0 : 0)

i that case client correctly trigger a panic as we have none errors in processing, but reply isn't filled correctly.
that bug should be affect isn't NFS only.

Comment by Alexey Lyashkov [ 24/Oct/13 ]

Li,

may you look into MDT code to verify - why that error isn't returned correctly to the client?
from my point view it's should be addressed to the

#if 0
        /* XXX is raw_lookup possible as intent operation? */
        if (rc != 0) {
                if (rc == -ENOENT)
                        mdt_set_disposition(info, ldlm_rep, DISP_LOOKUP_NEG);
                RETURN(rc);
        } else
                mdt_set_disposition(info, ldlm_rep, DISP_LOOKUP_POS);

        repbody = req_capsule_server_get(info->mti_pill, &RMF_MDT_BODY);
#endif

or we need to replace an 'RETURN(1);' to "return(rc)' at end of mdt_raw_lookup() function.

Comment by Patrick Farrell (Inactive) [ 24/Oct/13 ]

At Alexey's request, we reproduced this.

Here's the procedure from our test engineer:

1) Mount lustre on NFS server

2) Start nfsserver daemon on NFS server

3) Export nfs (sudo /usr/sbin/exportfs -i -o rw,insecure,no_root_squash,no_subtree_check,fsid=538 *:/extlus )

4) Mount NFS on client (sudo mount perses-esl3:/extlus /tmp/lus)

5) Run test using /tmp/lus

Other than fsid=, the other options are just what we usually use when testing NFS internally.

Attaching logs shortly.

Comment by Patrick Farrell (Inactive) [ 24/Oct/13 ]

MDS log during the test. Client LBUGged doing unlink8 test from LTP as described earlier.

Comment by Li Xi (Inactive) [ 25/Oct/13 ]

Hi Alexey,

I agree on that mdt_raw_lookup() should not return 1 all the time. And follwoing patch tries to fix that too.
http://review.whamcloud.com/#/c/7327

Comment by Alexey Lyashkov [ 25/Oct/13 ]

Hi Li,

main question for it - did we need a set intent disposition in reply. may you check - how it send from client? via mdc_intent_lock or other way ?

Comment by Patrick Farrell (Inactive) [ 28/Oct/13 ]

It might be worth noting that we hit this on 2.4.1. The ticket only lists 1.8.9/2.1.5.

Comment by Alexey Lyashkov [ 14/Nov/13 ]

any ability to answer ?

Comment by Li Xi (Inactive) [ 14/Nov/13 ]

Hi Alexey,

I am sorry, maybe because the lack of background knowledge, I don't understand the question well. Would you please explain a little bit about it? And do you have any specific problems about the patch?

Comment by Shuichi Ihara (Inactive) [ 14/Feb/14 ]

Alexey, can you please describe your quesiton in detial here?

Comment by Frederik Ferner (Inactive) [ 21/Aug/14 ]

Looks like we've just hit this as well on a NFS server/lustre client which is still running 1.8.9 after upgrading one file system to 2.5.2. We intend to upgrade the client to 2.5.2 as well ASAP but need to upgrade all file system first.

Is there any indication that this might be fixed in 2.5.2?

Comment by Patrick Farrell (Inactive) [ 22/Aug/14 ]

Frederik - There's no movement towards a fix at the moment. If you're building your own Lustre, there's an option: Alexey and Oleg dislike http://review.whamcloud.com/#/c/7327/, but it does avoid the bug & we've been running it at Cray for a bit.

Comment by Li Xi (Inactive) [ 01/Dec/14 ]

Patch of 'LU-3952 nfs: don't panic NFS server if MDS fails to find FID' helps to walk around the problem for master branch. But that patch does not help earlier versions such as b2_1. And I don't think the root cause has been fixed by that patch. Refreshed http://review.whamcloud.com/#/c/7327/ again.

Comment by Gerrit Updater [ 26/Dec/14 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/7327/
Subject: LU-3727 nfs: Fix ll_get_parent() LBUG caused by permission
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a0b959c53d10bf3f0fd6b22de46397d0c7e5f667

Comment by Gerrit Updater [ 07/Jan/15 ]

Lai Siyao (lai.siyao@intel.com) uploaded a new patch: http://review.whamcloud.com/13270
Subject: LU-3727 nfs: Fix ll_get_parent() LBUG caused by permission
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: cffa66fed624973d71bfc2c3382f1ef0a19397d4

Comment by Peter Jones [ 10/Feb/15 ]

Landed for 2.7

Comment by Gerrit Updater [ 20/Apr/15 ]

Lai Siyao (lai.siyao@intel.com) uploaded a new patch: http://review.whamcloud.com/14498
Subject: LU-3727 nfs: Fix ll_get_parent() LBUG caused by permission
Project: fs/lustre-release
Branch: b2_3
Current Patch Set: 1
Commit: bc90ead229cdaa940cca1819fe6fbfb983506014

Generated at Sat Feb 10 01:36:24 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.