[LU-6134] lfs find --size opens everything Created: 19/Jan/15  Updated: 05/Aug/20  Resolved: 06/Jul/15

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.7.0
Fix Version/s: Lustre 2.8.0

Type: Bug Priority: Minor
Reporter: John Hammond Assignee: Di Wang
Resolution: Fixed Votes: 0
Labels: lfs, mq115

Issue Links:
Duplicate
duplicates LU-1738 lfs find cannot find files in OST tha... Resolved
is duplicated by LU-4296 lfs find --ost if the OST immediatell... Resolved
Related
Severity: 3
Rank (Obsolete): 17093

 Description    
q:lustre-release# git describe
2.6.92-55-g2bc5bcb
q:lustre-release# unset MDSCOUNT
q:lustre-release# llmount.sh
...
q:lustre-release# cd /mnt/lustre
q:lustre# mkdir d0
q:lustre# touch d0/f{0..1023}
q:lustre# strace -o /tmp/1 lfs find --size +4K .
./d0
q:lustre-release# awk -F'(' '{ print $1; }' /tmp/1 | sort | uniq -c
      1 access
      1 arch_prctl
      3 brk
   1034 close
      1 execve
      1 exit_group
      1 fcntl
      8 fstat
      4 getdents
      1 gettimeofday
   3077 ioctl
   1024 lstat
     20 mmap
      9 mprotect
      2 munmap
   1034 open
      6 read
      1 write
q:lustre-release# grep ioctl /tmp/1 | sort | uniq -c
      1 ioctl(3, 0x800466af, 0x7ffff9778e00)    = 0
      1 ioctl(3, 0xc0086683, 0x7ffff9778890)    = 0
      1 ioctl(3, 0xc0086917, 0x2354040)         = 0
      1 ioctl(4, 0x800466af, 0x7ffff9778e00)    = 0
   1024 ioctl(4, 0xc00866a5, 0x2354040)         = 0 # IOC_LOV_GETINFO
   1024 ioctl(4, 0xc0086916, 0x2354040)         = 0 # IOC_MDC_GETFILEINFO
      1 ioctl(4, 0xc0086917, 0x2354040)         = 0 # LL_IOC_MDC_GETINFO
   1024 ioctl(5, 0x800466af, 0x7ffff9778e00)    = 0 # LL_IOC_GET_MDTIDX

For each file returned by getdents on d0 we do:

ioctl(4, 0xc0086916, 0x2354040)         = 0 # IOC_MDC_GETFILEINFO
lstat("./d0/f831", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
open("./d0/f831", O_RDONLY)             = 5
ioctl(5, 0x800466af, 0x7ffff9778e00)    = 0 # LL_IOC_GET_MDTIDX
close(5)                                = 0
ioctl(4, 0xc00866a5, 0x2354040)         = 0 # IOC_LOV_GETINFO


 Comments   
Comment by Andreas Dilger [ 19/Jan/15 ]

Di, it looks like this was landed back in 2.2 in commit 322968ac "LU-819 utils: Fix lfs getstripe -M" (http://review.whamcloud.com/1646) in order to implement the lfs find --mdt. However, if lfs find isn't actually using this option then it doesn't need to get the MDT index at all. Opening and closing each file adds a significant amount of overhead to this code that could easily be avoided.

Comment by Andreas Dilger [ 20/Feb/15 ]

It seems that this is also preventing lfs find --ost from working when an OST is offline, since the added fstat() will fail if the file is on an OST that is offline (LU-1738), which is one of the main reasons to use lfs find --ost in the first place. Will push a patch shortly.

Comment by Gerrit Updater [ 20/Feb/15 ]

Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: http://review.whamcloud.com/13822
Subject: LU-6134 utils: lfs should only open/stat files if needed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 363e65f2080aa6cd9b91e3c99d470528428e81df

Comment by Gerrit Updater [ 18/Mar/15 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/13822/
Subject: LU-6134 utils: lfs should only open/stat files if needed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9c4783a744f27da813b9c5be9c530f7772eec203

Comment by Peter Jones [ 06/Jul/15 ]

Landed for 2.8

Generated at Sat Feb 10 01:57:30 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.