https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3727 Lustre <p>At GE Global Research, we ran into an LBUG with a 1.8.9 client that is re-exporting 2.1.5 Lustre:</p> <p>Jul 31 10:26:46 scinfra3 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).<br/> Jul 31 10:26:46 scinfra3 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory<br/> Jul 31 10:26:46 scinfra3 kernel: NFSD: starting 90-second grace period<br/> Jul 31 10:26:53 scinfra3 ntpd<span class="error">&#91;8318&#93;</span>: synchronized to 3.40.208.30, stratum 2<br/> Jul 31 10:29:46 scinfra3 kernel: LustreError: 27396:0:(llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed<br/> Jul 31 10:29:46 scinfra3 kernel: LustreError: 27396:0:(llite_nfs.c:281:ll_get_parent()) LBUG<br/> Jul 31 10:29:46 scinfra3 kernel: Pid: 27396, comm: nfsd<br/> Jul 31 10:29:46 scinfra3 kernel: <br/> Jul 31 10g:29:46 scinfra3 kernel: Call Trace:<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] libcfs_debug_dumpstack+0x51/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] lbug_with_loc+0x7a/0xd0 <span class="error">&#91;libcfs&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] tracefile_init+0x0/0x110 <span class="error">&#91;libcfs&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] ll_get_parent+0x1e3/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] ll_get_dentry+0x6b/0xe0 <span class="error">&#91;lustre&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] mutex_lock+0xd/0x1d<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] find_exported_dentry+0x241/0x486 <span class="error">&#91;exportfs&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_acceptable+0x0/0xdc <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] autoremove_wake_function+0x0/0x2e<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] sunrpc_cache_lookup+0x4b/0x128 <span class="error">&#91;sunrpc&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] exp_get_by_name+0x5b/0x71 <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] exp_find_key+0x89/0x9c <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_acceptable+0x0/0xdc <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] ll_decode_fh+0x197/0x240 <span class="error">&#91;lustre&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] set_current_groups+0x116/0x164<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] fh_verify+0x29c/0x4cf <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd3_proc_getattr+0x8a/0xbe <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd_dispatch+0xd8/0x1d6 <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] svc_process+0x3f8/0x6bf <span class="error">&#91;sunrpc&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] __down_read+0x12/0x92<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x1a5/0x2cb <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] child_rip+0xa/0x11<br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] nfsd+0x0/0x2cb <span class="error">&#91;nfsd&#93;</span><br/> Jul 31 10:29:46 scinfra3 kernel: [ ] child_rip+0x0/0x11<br/> Jul 31 10:29:46 scinfra3 kernel: </p> <p>It appears to be easily reproducible, we are going to try to get a core dump, but I was wondering if there was anything obvious from this trace or any other jira tickets I might have missed. Also is there any other information that might be useful?</p> <p>Thanks.</p> LU-3727

LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body->valid & OBD_MD_FLID) failed

Bug Minor Resolved Fixed Emoly Liu Oz Rentas patch Thu, 8 Aug 2013 17:53:18 +0000 Thu, 14 Jun 2018 21:41:36 +0000 Tue, 10 Feb 2015 12:39:17 +0000 Lustre 2.1.5 Lustre 1.8.9 Lustre 2.4.1 Lustre 2.7.0 0 14 <p>Thanks for the report Kit!</p> <p>Emoly</p> <p>What do you suggest here?</p> <p>Peter</p> <p>Kit, could you please show how to reproduce this LBUG in detail? And a core dump file will be helpful. Thanks!</p> <p>Hi Emoly,</p> <p>The customer is currently unable to reproduce after setting up kdump, so we are in a holding pattern. I will ask what they were doing to reproduce before.</p> <p>Thanks. </p> <p>Hi Emoly,</p> <p>We were able to capture a crash dump from a different client node. Strangely, this client gets a null pointer dereference when trying to print the LBUG stack trace. </p> <p>Here is the vmcore:<br/> <a href="http://ddntsr.com/ftp/2013-08-13-SR23991-vmcore" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2013-08-13-SR23991-vmcore</a></p> <p>This is the kernel debuginfo for it:<br/> <a href="http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-279.2.1.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-279.2.1.el6.x86_64.rpm</a><br/> <a href="http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-279.2.1.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-279.2.1.el6.x86_64.rpm</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; bt PID: 23553 TASK: ffff881019a32080 CPU: 10 COMMAND: <span class="code-quote">"nfsd"</span> #0 [ffff88100a241520] machine_kexec at ffffffff8103281b #1 [ffff88100a241580] crash_kexec at ffffffff810ba792 #2 [ffff88100a241650] text_poke at ffffffff815016f0 #3 [ffff88100a241680] no_context at ffffffff81043bab #4 [ffff88100a2416d0] __bad_area_nosemaphore at ffffffff81043e35 #5 [ffff88100a241720] bad_area_nosemaphore at ffffffff81043f03 #6 [ffff88100a241730] __do_page_fault at ffffffff81044661 #7 [ffff88100a241850] debugfs_kprobe_init at ffffffff815036ce #8 [ffff88100a241880] do_debug at ffffffff81500a85 #9 [ffff88100a2419d8] libcfs_debug_dumpstack at ffffffffa01d78f5 [libcfs] #10 [ffff88100a2419f8] lbug_with_loc at ffffffffa01d7f25 [libcfs] #11 [ffff88100a241a48] libcfs_assertion_failed at ffffffffa01e0696 [libcfs] #12 [ffff88100a241a98] ll_get_parent at ffffffffa08836f8 [lustre] #13 [ffff88100a241b38] reconnect_path at ffffffffa021e3b0 [exportfs] #14 [ffff88100a241ba8] exportfs_decode_fh at ffffffffa021e7aa [exportfs] #15 [ffff88100a241d18] fh_verify at ffffffffa064abea [nfsd] #16 [ffff88100a241da8] nfsd3_proc_getattr at ffffffffa0655b6c [nfsd] #17 [ffff88100a241dd8] nfsd_dispatch at ffffffffa064743e [nfsd] #18 [ffff88100a241e18] svc_process_common at ffffffffa05fb5d4 [sunrpc] #19 [ffff88100a241e98] svc_process at ffffffffa05fbc10 [sunrpc] #20 [ffff88100a241eb8] nfsd at ffffffffa0647b62 [nfsd] #21 [ffff88100a241ee8] kthread at ffffffff81091d66 #22 [ffff88100a241f48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>The customer uses NFS for staging files onto the clusters. So it should just be a lot of copies, removes, that sort of thing. There shouldn't be any jobs, but it's possible there are some desktop applications that run against it. It happens pretty frequently, though they don't have a reproducer. </p> <p>Let me know if there is any other data I can get.</p> <p>Thanks.</p> <p>We hit the same problem. Here is how we reproduce it:<br/> 1. Create a directory under the Lustre directory which is exported, e.g. create /mnt/lustre/export/dir.<br/> 2. Change the directory's mode to 700 so that no one other than root has the right to access it.<br/> 3. Export the Lustre directry, i.e. /mnt/lustre/export. Please make sure 'no_root_squash' option is not enabled.<br/> 4. Mount the NFS client on some node.<br/> 5. On the NFS client, cd into the Lustre directory, i.e. directory of /mnt/lustre/export.<br/> 6. On the NFS server (and the Lustre client), unexport the NFS server, unmount Lustre client,<br/> 7. On the NFS client, run 'ls -l'. The operation will stuck.<br/> 8. On the NFS server, remount Lustre client, reexport NFS server (i.e. simulate a reboot).<br/> 9. Wait for a while, we will hit the LBUG (on master branch) or get EACCES (on Lustre-2.1.4)</p> <p>After some inverstigation, we found that the cause is that the nfs daemon user does not have the permission to access ".." directory.</p> <p>We will upload the fix patch soon.</p> <p>Here is the patch which tries to fix the problem.<br/> <a href="http://review.whamcloud.com/7327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7327</a></p> <p>Thanks for your patch!</p> <p>Here is a patch which tries to fix the problem in another way:<br/> <a href="http://review.whamcloud.com/#/c/7349/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7349/</a></p> <p>The last patch (<a href="http://review.whamcloud.com/7327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7327</a>) tries to fix the problem by skipping the permission check on MDT. This patch avoids permission denied by pretending the client is sending a RPC of root user. It is useful for us because this is a client-side-only patch, and it is difficult for us to get downtime for the customer.</p> <p>I am not sure which patch is better. Maybe we should think more about their security problem?</p> <p>I am not sure whether I understand your case correctly or not. But consider the following case:</p> <p>1) root user "mkdir /tmp/test"<br/> 2) root user "chmod 0700 /tmp/test"<br/> 3) root user "cd /tmp/test"<br/> 4) root user "su USER1"<br/> 5) USER1 "ls -l"</p> <p>The result is "ls: cannot open directory .: Permission denied".</p> <p>Back to the <a href="https://jira.whamcloud.com/browse/LU-3727" title="LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed" class="issue-link" data-issue-key="LU-3727"><del>LU-3727</del></a>, if the "/mnt/lustre/export" is only accessible for root user, then "ls -l" under it as non-root user should get "Permission denied" instead of bypassing permission check as the patch will do, right? If yes, we need to remove invalid "ASSERT()" and report "Permission denied" as expected.</p> <p>Sorry, maybe my former explaination was not accurate. Here is how to reproduce the problem.<br/> 1) NFS server "mkdir /mnt/lustre/export"<br/> 2) NFS server exports /mnt/lustre/export<br/> 3) NFS server "mkdir /mnt/lustre/export/dir"<br/> 4) NFS server "chmod 0700 /mnt/lustre/export/dir"<br/> 5) NFS client "cd /mnt/lustre/export/"<br/> 6) NFS server "service nfs stop"<br/> 7) NFS client "ls -l"<br/> 8) NFS server "umount /mnt/lustre"<br/> 9) NFS server mount /mnt/lustre again<br/> 10) NFS server "service nfs start"<br/> 11) NFS client hits LBUG after a few seconds</p> <p>Please notice that at step 7), we are under "/mnt/lustre/export", and we have the permission for this operation.<br/> If "no_root_squash" is disabled, the problem will show up no matter the user is root or not.</p> <p>Hi FanYong &amp; Emoly</p> <p>Would you please review Li's patch quickly? <br/> And please give us advices/comments on this, please?</p> <p>Thanks!</p> <p>I think the patch (<a href="http://review.whamcloud.com/#/c/7349/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7349/</a>) is some hack, any will introduce security hole. There may be UID/GID mapping on MDT side, so even if you specify the special case as root user, it still possible be mapped to another user in the future. It is MDT to decide how to handle such case.</p> <p>Thank you very much, Fan Yong!</p> <p>I guess the first patch (<a href="http://review.whamcloud.com/7327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7327</a>) is more acceptable, right?</p> <p>I think 7327 is better than 7349, although the former one still can be improved.</p> <p>Li Xi, </p> <p>I have some questions about you script to reproduce a bug.<br/> 1) if nfs client will able to do 'cd $dir' they should be lookup a permissions in last directory as minimum.<br/> so you should be have EACCESS in that step.<br/> 2) in case nfs client don't have any verify permissions in that step - have a EACCESS in return for ls -l command, looks reasonable, but you tried avoid that permission check and take output.</p> <p>if (2) is true - i think question - why 'md_getattr_name' isn't return an error as it's expected.</p> <p>may you attach a full lustre debug from lustre in that crash? i think you use a single node configuration for a nfs server so we will be see all operations in logs.</p> <p>The trace output when this BUG happens. Please note that this log is trace on Lustre-2.1, so no LBUG happens. But basically, it is the same problem.</p> <p>Hi Alexey,</p> <p>Yeah, NFS client has the permission to access '/mnt/lustre/export/' but it has no permission to access '/mnt/lustre/export/dir'. However, when NFS daemon restart (which is not normal and that is when ll_get_parent() is called), NFS client should get the attribute of '/mnt/lustre/export/dir/../', which will make Lustre client (and NFS server) hit the LBUG. Normally, NFS client could access '/mnt/lustre/export/' withouth any problem though it has no permission to access '/mnt/lustre/export/dir', but ll_get_parent() is different. Since there is no reason to require that a user doing 'ls -l /mnt/lustre/export/' has the permission to access '/mnt/lustre/export/dir', I think the best way to fix this is to avoid the permission check of '/mnt/lustre/export/dir' in this case.</p> <p>I've post the lustre debug file as 'log.txt'. Sorry, I should have post it earlier.</p> <p>We encountered this assertion while running a set of tests from the Linux Test Project over NFS. We did not do any unmount/remounting of NFS or Lustre as part of this, but are consistently able to hit the bug with this test. We hit it with both a SLES11SP1 Lustre client and a CentOS 6.4 Lustre client, in both cases to CentOS 6.4 servers running 2.4.1.</p> <p>I'll attached the source file for this particular test (the LTP is a GPL'ed project).</p> <p>The test set is called unlink08, and is a series of tests of the unlink system call.</p> <p>We're planning to test the patch in <a href="http://review.whamcloud.com/#/c/7327/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7327/</a> today, I'll report back with results.</p> <p>With 3727 applied, the 'unlink8' test no longer hits this bug.</p> <p><span class="error">&#91;root@rhel6-64 WC-review&#93;</span># gcc unlink08.c <br/> unlink08.c:119:18: error: test.h: No such file or directory<br/> unlink08.c:120:21: error: usctest.h: No such file or directory</p> <p>Sorry, Alexey - I hadn't intended that to be buildable/useable by itself, I just posted it for reference. </p> <p>It's part of LTP, which can be downloaded from here:<br/> <a href="http://sourceforge.net/projects/ltp/files/latest/download" class="external-link" target="_blank" rel="nofollow noopener">http://sourceforge.net/projects/ltp/files/latest/download</a></p> <p>You'll find it in testcases/kernel/syscalls/unlink/ in the untarred package, but you'll have to figure out how to build and run that specific test.</p> <p>I have checked on ~2 days ago with some older ltp code, and don't able to replicate that assertion.<br/> but my test env hit an hung while accessing an exported dir.</p> <p>after hung finished </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@rhel6-64 ltp]# export TMPDIR=/mnt/lustre2/; /Users/shadow/work/lustre/work/ltp/testcases/kernel/syscalls/unlink/unlink08 unlink08 1 TPASS : unlink(&lt;unwritable directory&gt;) returned 0 unlink08 2 TPASS : unlink(&lt;unsearchable directory&gt;) returned 0 unlink08 3 TPASS : unlink(&lt;directory&gt;) Failed, errno=21 </pre> </div></div> <p>where </p> <ol> <li>cat /etc/exports<br/> /mnt/lustre/export 192.168.0.0/16(rw,no_root_squash,fsid=100)</li> </ol> <p>/without fsid i have issues with exporting directory while root lustre dir exported fine/</p> <p>ps. same for last LTP from git.</p> <p>Interesting. I confirmed and we were able to cause that assertion by running only unlink8, as well as running unlink8 as part of the larger test suite.<br/> No idea why we're seeing it and you aren't. Regardless, as mentioned above, the patch fixes that issue for us.</p> <p>By the way...<br/> You may already know the details on the FSID issue, but if not, they're here:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-3550" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-3550</a><br/> Very briefly, Lustre 2.4 uses 64 bit inodes. NFS does not support 64 bit root inodes, because it explicitly casts inodes to 32 bit in FSID generation, and when those generated FSIDs are returned by the client and unparsed by the server, they cannot be matched back to the 64 bit root inode on Lustre. (In contrast, when you provide an FSID with -o fsid, it is used directly, so this issue is avoided. Why a generated FSID is parsed to an inode and looked up by the server - rather than used as an arbitrary ID for the export, like when an FSID is provided - is something of a mystery.)</p> <p>question about fsid - is long story. lustre put native fid with parent fid in NFS file handle, FSID may be any number and it's now same as block device id (for compatibility with older servers in pair). but main question - why i able to export /mnt/lustre without fsid set, but /mnt/lustre/export need fsid.</p> <p>as about assert - did you able to take a crashdump and extract lustre debug log from core file?<br/> and provide a git hash of lustre sources.</p> <p>re FSID:<br/> I'm guessing, but I believe because /mnt/lustre has an inode # in your normal Linux file system, whereas /mnt/lustre/export exists only in Lustre. <br/> It may matter if you mount Lustre before or after exporting /mnt/lustre - Possibly if you mount lustre at /mnt/lustre/ first and then do the exportfs, you would get the Lustre inode # and see the problem. <br/> I recall having problems exporting the root of Lustre mounts as well as sub-directories of those mounts, but I was definitely mounting Lustre before doing the export.</p> <p>re: The assertion.<br/> We were running Cray's version of Lustre 2.4.1, so no Git hash. However, it's very close to the Intel 2.4.1 tag.</p> <p>We could probably extract logs from the dump we have, but the logs only had the default debugging options, IE:<br/> debug=ioctl neterror warning error emerg ha config console</p> <p>If you're still interested, I could probably get those for you tomorrow.</p> <p>it's don't enough <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/>... i need full debug logs for both - mdt and client nodes.. primary for a client, but mdt also interested. I will try checkout to 2.4.1 and see a difference.</p> <p>tried to replicate with v2_4_1 tag, but without success.</p> <p>Li,</p> <p>attached log don't have an information about rpc and isn't full - it's have just last step when we hit an error..</p> <p>I reproduced the problem with following steps.<br/> 1. Run all the servers/client on the same machine.<br/> <span class="error">&#91;root@server1 ~&#93;</span># df<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> /dev/sda3 100798068 92258100 3419652 97% /<br/> none 16424096 0 16424096 0% /dev/shm<br/> /dev/sda1 521780 216388 278888 44% /boot<br/> /dev/sda5 1043548 43396 947140 5% /mnt/mgs<br/> /dev/sda6 7865980 456800 6885408 7% /mnt/mdt0<br/> /dev/sda7 7865980 456396 6885812 7% /mnt/mdt1<br/> /dev/sda8 33442176 562620 31201168 2% /mnt/ost0<br/> /dev/sda9 80025512 464196 75545068 1% /mnt/ost1<br/> 192.168.3.100@tcp:/server1<br/> 113467688 1026816 106746236 1% /mnt/lustre<br/> 2. NFS server is on the same machine. Please note no_root_squash should NOt be enabled to reproduce this problem using root<br/> <span class="error">&#91;root@server1 ~&#93;</span># cat /etc/exports <br/> /mnt/lustre/nfs *(fsid=0,rw)<br/> 3. The directory should have no execute permission for normal users.<br/> <span class="error">&#91;root@server1 ~&#93;</span># ll /mnt/lustre/nfs/<br/> total 4<br/> drwx------ 2 root root 4096 Oct 24 11:39 dir<br/> 4. Trace the path<br/> <span class="error">&#91;root@server1 ~&#93;</span># echo trace &gt; /proc/sys/lnet/debug<br/> 5. Clearup the messages<br/> <span class="error">&#91;root@server1 ~&#93;</span># lctl debug_kernel /tmp/lustre.log<br/> 6. Start the NFS<br/> <span class="error">&#91;root@server1 ~&#93;</span># service nfs restart<br/> 7. Start the NFS client<br/> <span class="error">&#91;root@server1 ~&#93;</span># mount 192.168.3.100:/mnt/lustre/nfs /mnt/nfs<br/> 8. Go down to the NFS client directory<br/> <span class="error">&#91;root@server1 ~&#93;</span># cd /mnt/nfs<br/> 9. Get the target dentry so that ll_get_parent will call for it.<br/> <span class="error">&#91;root@server1 nfs&#93;</span># ls -l<br/> 10. Get the target dentry so that ll_get_parent will call for it.<br/> 11. Stop the NFS server to simulate a reboot<br/> <span class="error">&#91;root@server1 nfs&#93;</span># service nfs stop<br/> Shutting down NFS daemon: [ OK ]<br/> Shutting down NFS mountd: [ OK ]<br/> Shutting down NFS quotas: [ OK ]<br/> Shutting down NFS services: [ OK ]<br/> <span class="error">&#91;root@server1 nfs&#93;</span># umount /mnt/lustre<br/> 12. Access the directory again, this command will stuck<br/> <span class="error">&#91;root@server1 nfs&#93;</span># ls -l<br/> 13. Start the NFS server again on another terminal<br/> <span class="error">&#91;root@server1 ~&#93;</span># mount -t lustre 192.168.3.100@tcp:/server1 /mnt/lustre<br/> <span class="error">&#91;root@server1 ~&#93;</span># service nfs start<br/> WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.<br/> Starting NFS services: [ OK ]<br/> Starting NFS quotas: [ OK ]<br/> Starting NFS mountd: [ OK ]<br/> Stopping RPC idmapd: [ OK ]<br/> Starting RPC idmapd: [ OK ]<br/> Starting NFS daemon: [ OK ]<br/> 14. After a few seconds the stucked command hit the LBUG.<br/> <span class="error">&#91;root@server1 nfs&#93;</span># ls -l</p> <p>Message from syslogd@server1 at Oct 24 12:08:53 ...<br/> kernel:LustreError: 3348:0:(llite_nfs.c:311:ll_get_parent()) ASSERTION( body-&gt;valid &amp; (0x00000001ULL) ) failed: </p> <p>Message from syslogd@server1 at Oct 24 12:08:53 ...<br/> kernel:LustreError: 3348:0:(llite_nfs.c:311:ll_get_parent()) LBUG<br/> 15. Get the Lustre log<br/> lctl debug_file /tmp/lustre-log.1382641733.3348 /tmp/lustre.log </p> <p>Alexey,</p> <p>'lustre.log' is the trace log which I got when reproducing the problem. Hope it helps.</p> <p>Li,</p> <p>thanks!</p> <p>Li,</p> <p>thanks again. devil in details.. we need additional directory created in exported dir.<br/> without it ll isn't trigger a bug.</p> <p>as i talk before - mdt generate an error as before</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:1.0:1382635559.670116:0:15672:0:(mdd_permission.c:309:__mdd_permission_internal()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3) 00000004:00000001:1.0:1382635559.670117:0:15672:0:(mdd_dir.c:90:__mdd_lookup()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3) 00000004:00000001:1.0:1382635559.670117:0:15672:0:(mdd_dir.c:115:mdd_lookup()) Process leaving (rc=18446744073709551603 : -13 : fffffffffffffff3) </pre> </div></div> <p>but that error isn't returned to the caller</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:1.0:1382635559.670119:0:15672:0:(mdt_handler.c:1273:mdt_getattr_name_lock()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>i that case client correctly trigger a panic as we have none errors in processing, but reply isn't filled correctly.<br/> that bug should be affect isn't NFS only.</p> <p>Li,</p> <p>may you look into MDT code to verify - why that error isn't returned correctly to the client?<br/> from my point view it's should be addressed to the</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#<span class="code-keyword">if</span> 0 <span class="code-comment">/* XXX is raw_lookup possible as intent operation? */</span> <span class="code-keyword">if</span> (rc != 0) { <span class="code-keyword">if</span> (rc == -ENOENT) mdt_set_disposition(info, ldlm_rep, DISP_LOOKUP_NEG); RETURN(rc); } <span class="code-keyword">else</span> mdt_set_disposition(info, ldlm_rep, DISP_LOOKUP_POS); repbody = req_capsule_server_get(info-&gt;mti_pill, &amp;RMF_MDT_BODY); #endif </pre> </div></div> <p>or we need to replace an 'RETURN(1);' to "return(rc)' at end of mdt_raw_lookup() function.</p> <p>At Alexey's request, we reproduced this.</p> <p>Here's the procedure from our test engineer:<br/> &#8212;<br/> 1) Mount lustre on NFS server</p> <p>2) Start nfsserver daemon on NFS server</p> <p>3) Export nfs (sudo /usr/sbin/exportfs -i -o rw,insecure,no_root_squash,no_subtree_check,fsid=538 &#42;:/extlus )</p> <p>4) Mount NFS on client (sudo mount perses-esl3:/extlus /tmp/lus)</p> <p>5) Run test using /tmp/lus<br/> &#8212;<br/> Other than fsid=, the other options are just what we usually use when testing NFS internally.</p> <p>Attaching logs shortly.</p> <p>MDS log during the test. Client LBUGged doing unlink8 test from LTP as described earlier.</p> <p>Hi Alexey,</p> <p>I agree on that mdt_raw_lookup() should not return 1 all the time. And follwoing patch tries to fix that too.<br/> <a href="http://review.whamcloud.com/#/c/7327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7327</a></p> <p>Hi Li,</p> <p>main question for it - did we need a set intent disposition in reply. may you check - how it send from client? via mdc_intent_lock or other way ?</p> <p>It might be worth noting that we hit this on 2.4.1. The ticket only lists 1.8.9/2.1.5.</p> <p>any ability to answer ?</p> <p>Hi Alexey,</p> <p>I am sorry, maybe because the lack of background knowledge, I don't understand the question well. Would you please explain a little bit about it? And do you have any specific problems about the patch?</p> <p>Alexey, can you please describe your quesiton in detial here?</p> <p>Looks like we've just hit this as well on a NFS server/lustre client which is still running 1.8.9 after upgrading one file system to 2.5.2. We intend to upgrade the client to 2.5.2 as well ASAP but need to upgrade all file system first. </p> <p>Is there any indication that this might be fixed in 2.5.2?</p> <p>Frederik - There's no movement towards a fix at the moment. If you're building your own Lustre, there's an option: Alexey and Oleg dislike <a href="http://review.whamcloud.com/#/c/7327/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7327/</a>, but it does avoid the bug &amp; we've been running it at Cray for a bit.</p> <p>Patch of '<a href="https://jira.whamcloud.com/browse/LU-3952" title="llite_nfs.c:349:ll_get_parent()) ASSERTION( body-&gt;valid &amp; (0x00000001ULL) ) failed" class="issue-link" data-issue-key="LU-3952"><del>LU-3952</del></a> nfs: don't panic NFS server if MDS fails to find FID' helps to walk around the problem for master branch. But that patch does not help earlier versions such as b2_1. And I don't think the root cause has been fixed by that patch. Refreshed <a href="http://review.whamcloud.com/#/c/7327/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7327/</a> again.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/7327/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7327/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3727" title="LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed" class="issue-link" data-issue-key="LU-3727"><del>LU-3727</del></a> nfs: Fix ll_get_parent() LBUG caused by permission<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a0b959c53d10bf3f0fd6b22de46397d0c7e5f667</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13270" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13270</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3727" title="LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed" class="issue-link" data-issue-key="LU-3727"><del>LU-3727</del></a> nfs: Fix ll_get_parent() LBUG caused by permission<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: cffa66fed624973d71bfc2c3382f1ef0a19397d4</p> <p>Landed for 2.7</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14498" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14498</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3727" title="LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed" class="issue-link" data-issue-key="LU-3727"><del>LU-3727</del></a> nfs: Fix ll_get_parent() LBUG caused by permission<br/> Project: fs/lustre-release<br/> Branch: b2_3<br/> Current Patch Set: 1<br/> Commit: bc90ead229cdaa940cca1819fe6fbfb983506014</p> Related LU-5730 Development Rank 1|hzvxgf: Rank (Obsolete) 9597 Severity