[LU-12918] PCC should tolerate backend running out of inodes

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.13.0
• Labels: PCC
• Assignee: Qian Yingjin
• Reporter: James Nunez (Inactive)

Description

When the space of the local PCC file system (backend) is exhausted, PCC-RW falls back to the normal I/O path automatically since the capacity of the Lustre file system is probably much larger than the capacity of the PCC device. In this case, the user may notice a pause in writing, but all operations should complete successfully.

PCC should tolerate running out of inodes on the local PCC file system (backend) and continue the operation by writing/creating files directly to the Lustre file system. Currently, the application gets an error when the PCC backend runs out of inodes. I’ve seen this issue with auto attach and without auto attach.

In the following example, /mnt/pcc is a small local ext4 file system on a client node.

# mkdir /lustre/scratch/lpcc
# lfs project -sp 100 /lustre/scratch/lpcc
# lctl pcc add /lustre/scratch/ /mnt/pcc/ -p "projid={100} rwid=2 open_attach=1"
# lctl pcc list /lustre/scratch/
/mnt/pcc/:
  rwid: 2
  flags: 7
  autocache: projid={100}
# lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
scratch-MDT0000_UUID    26213000       77904    23777136   1% /lustre/scratch[MDT:0]
scratch-OST0000_UUID    45430936       54292    43000964   1% /lustre/scratch[OST:0]
scratch-OST0001_UUID    45430936     3202076    39853180   8% /lustre/scratch[OST:1]

filesystem_summary:     90861872     3256368    82854144   4% /lustre/scratch

# df /mnt/pcc
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vda3        1998672  6416   1871016   1% /mnt/pcc
# compilebench -D /lustre/scratch/lpcc -i 2 -r 2
using working directory /lustre/scratch/lpcc, 2 intial dirs 2 runs
Traceback (most recent call last):
  File "/usr/bin/compilebench", line 567, in <module>
    dset = dataset(options.sources, rnd)
  File "/usr/bin/compilebench", line 319, in _init_
    self.unpatched = native_order(self.unpatched, "unpatched")
  File "/usr/bin/compilebench", line 97, in native_order
    run_directory(tmplist, dirname, "native %s" % tag)
  File "/usr/bin/compilebench", line 225, in run_directory
    fp = file(fname, 'a+')
IOError: [Errno 28] No space left on device: '/lustre/scratch/lpcc/native-0/include/linux/font.h'

# df /mnt/pcc
Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/vda3        1998672 695880   1181552  38% /mnt/pcc
# df -i /mnt/pcc
Filesystem     Inodes  IUsed IFree IUse% Mounted on
/dev/vda3      131072 131072     0  100% /mnt/pcc

# lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
scratch-MDT0000_UUID    26213000       82676    23772364   1% /lustre/scratch[MDT:0]
scratch-OST0000_UUID    45430936       54292    43000964   1% /lustre/scratch[OST:0]
scratch-OST0001_UUID    45430936     3202076    39853180   8% /lustre/scratch[OST:1]

filesystem_summary:     90861872     3256368    82854144   4% /lustre/scratch