Details
-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Lustre 2.7.0
-
None
-
CentOS-6.8
lustre-2.7.2-2nasS_mofed32v3f_2.6.32_573.26.1.el6.20160517.x86_64.lustre272.x86_64
kernel-2.6.32-573.26.1.el6.20160517.x86_64.lustre272
-
3
-
9223372036854775807
Description
The read_cache_enable=1 and writethrough_cache_enable=1 settings don't appear to be retained through unmount/remount of the OSTs. However, the readcache_max_filesize=4194304 DOES get retained.
Is there a step I'm missing? Is there additional debugging procedures I should follow to trace down the source of the problem?
Example commands showing that the setting does get propagated after a "lctl conf_param", but then goes away...
nbp7-mds2 ~ # for i in $(lctl list_param osc.fscache*OST* | sed 's/osc\.//; s/-osc-.*//'); do lctl conf_param $i.ost.read_cache_enable=1; lctl conf_param $i.ost.writethrough_cache_enable=1; lctl conf_param $i.ost.readcache_max_filesize=4M; done
(wait ~10 seconds to propagate)
Nothing unusual in the OSS logs. In the logs on the MGS:
Dec 16 14:28:26 nbp7-mds2 kernel: Lustre: Modifying parameter fscache-OST0005.ost.read_cache_enable in log fscache-OST0005 Dec 16 14:28:26 nbp7-mds2 kernel: Lustre: Skipped 17 previous similar messages
nbp1-oss6 ~ # lctl get_param obdfilter.fscache*.{*_cache_enable,readcache_max_filesize}
obdfilter.fscache-OST0005.read_cache_enable=1
obdfilter.fscache-OST0005.writethrough_cache_enable=1
obdfilter.fscache-OST000b.read_cache_enable=1
obdfilter.fscache-OST000b.writethrough_cache_enable=1
obdfilter.fscache-OST0011.read_cache_enable=1
obdfilter.fscache-OST0011.writethrough_cache_enable=1
obdfilter.fscache-OST0017.read_cache_enable=1
obdfilter.fscache-OST0017.writethrough_cache_enable=1
obdfilter.fscache-OST001d.read_cache_enable=1
obdfilter.fscache-OST001d.writethrough_cache_enable=1
obdfilter.fscache-OST0023.read_cache_enable=1
obdfilter.fscache-OST0023.writethrough_cache_enable=1
obdfilter.fscache-OST0005.readcache_max_filesize=4194304
obdfilter.fscache-OST000b.readcache_max_filesize=4194304
obdfilter.fscache-OST0011.readcache_max_filesize=4194304
obdfilter.fscache-OST0017.readcache_max_filesize=4194304
obdfilter.fscache-OST001d.readcache_max_filesize=4194304
obdfilter.fscache-OST0023.readcache_max_filesize=4194304
# lmount -u -v -f fscache.csv --host service636 umounting OSTs... ssh -n service636 umount /mnt/lustre/OST29 ssh -n service636 umount /mnt/lustre/OST5 ssh -n service636 umount /mnt/lustre/OST35 ssh -n service636 umount /mnt/lustre/OST17 ssh -n service636 umount /mnt/lustre/OST11 ssh -n service636 umount /mnt/lustre/OST23
nbp1-oss6 ~ # lctl get_param obdfilter.fscache*.{*_cache_enable,readcache_max_filesize}
error: get_param: obdfilter/fscache*/*_cache_enable: Found no match
error: get_param: obdfilter/fscache*/readcache_max_filesize: Found no match
# lmount -m -v -f fscache.csv --host service636 mounting OSTs... ssh -n service636 'mkdir -p /mnt/lustre/OST29 ; mount -t lustre $(journal-dev-of.sh dev/intelcas1-29) -o errors=panic,extents,mballoc /dev/intelcas1-29 /mnt/lustre/OST29' ssh -n service636 'mkdir -p /mnt/lustre/OST5 ; mount -t lustre $(journal-dev-of.sh /dev/intelcas1-5) -o errors=panic,extents,mballoc /dev/intelcas1-5 /mnt/lustre/OST5' ssh -n service636 'mkdir -p /mnt/lustre/OST35 ; mount -t lustre $(journal-dev-of.sh /dev/intelcas2-35) -o errors=panic,extents,mballoc /dev/intelcas2-35 /mnt/lustre/OST35' ssh -n service636 'mkdir -p /mnt/lustre/OST17 ; mount -t lustre $(journal-dev-of.sh /dev/intelcas1-17) -o errors=panic,extents,mballoc /dev/intelcas1-17 /mnt/lustre/OST17' ssh -n service636 'mkdir -p /mnt/lustre/OST11 ; mount -t lustre $(journal-dev-of.sh /dev/intelcas2-11) -o errors=panic,extents,mballoc /dev/intelcas2-11 /mnt/lustre/OST11' ssh -n service636 'mkdir -p /mnt/lustre/OST23 ; mount -t lustre $(journal-dev-of.sh /dev/intelcas2-23) -o errors=panic,extents,mballoc /dev/intelcas2-23 /mnt/lustre/OST23'
# lctl get_param obdfilter.fscache*.{*_cache_enable,readcache_max_filesize}
obdfilter.fscache-OST0005.read_cache_enable=0
obdfilter.fscache-OST0005.writethrough_cache_enable=0
obdfilter.fscache-OST000b.read_cache_enable=0
obdfilter.fscache-OST000b.writethrough_cache_enable=0
obdfilter.fscache-OST0011.read_cache_enable=0
obdfilter.fscache-OST0011.writethrough_cache_enable=0
obdfilter.fscache-OST0017.read_cache_enable=0
obdfilter.fscache-OST0017.writethrough_cache_enable=0
obdfilter.fscache-OST001d.read_cache_enable=0
obdfilter.fscache-OST001d.writethrough_cache_enable=0
obdfilter.fscache-OST0023.read_cache_enable=0
obdfilter.fscache-OST0023.writethrough_cache_enable=0
obdfilter.fscache-OST0005.readcache_max_filesize=4194304
obdfilter.fscache-OST000b.readcache_max_filesize=4194304
obdfilter.fscache-OST0011.readcache_max_filesize=4194304
obdfilter.fscache-OST0017.readcache_max_filesize=4194304
obdfilter.fscache-OST001d.readcache_max_filesize=4194304
obdfilter.fscache-OST0023.readcache_max_filesize=4194304
I should note that the MDS does have a single MGS for two file systems, in case that is relevant to reproducing the problem...
nbp7-mds2 ~ # lctl dl 0 UP osd-ldiskfs MGS-osd MGS-osd_UUID 5 1 UP mgs MGS MGS 9 2 UP mgc MGC10.151.27.39@o2ib 2eb7f880-6a2e-ea5e-3631-922183627327 5 3 UP osd-ldiskfs nocache-MDT0000-osd nocache-MDT0000-osd_UUID 13 4 UP mds MDS MDS_uuid 3 5 UP lod nocache-MDT0000-mdtlov nocache-MDT0000-mdtlov_UUID 4 6 UP mdt nocache-MDT0000 nocache-MDT0000_UUID 19 7 UP mdd nocache-MDD0000 nocache-MDD0000_UUID 4 8 UP qmt nocache-QMT0000 nocache-QMT0000_UUID 4 9 UP osp nocache-OST0029-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 10 UP osp nocache-OST002f-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 11 UP osp nocache-OST003b-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 12 UP osp nocache-OST0035-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 13 UP osp nocache-OST0041-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 14 UP osp nocache-OST0047-osc-MDT0000 nocache-MDT0000-mdtlov_UUID 5 15 UP lwp nocache-MDT0000-lwp-MDT0000 nocache-MDT0000-lwp-MDT0000_UUID 5 16 UP osd-ldiskfs fscache-MDT0000-osd fscache-MDT0000-osd_UUID 13 17 UP lod fscache-MDT0000-mdtlov fscache-MDT0000-mdtlov_UUID 4 18 UP mdt fscache-MDT0000 fscache-MDT0000_UUID 17 19 UP mdd fscache-MDD0000 fscache-MDD0000_UUID 4 20 UP qmt fscache-QMT0000 fscache-QMT0000_UUID 4 21 UP osp fscache-OST0023-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 22 UP osp fscache-OST0011-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 23 UP osp fscache-OST000b-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 24 UP osp fscache-OST001d-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 25 UP osp fscache-OST0017-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 26 UP osp fscache-OST0005-osc-MDT0000 fscache-MDT0000-mdtlov_UUID 5 27 UP lwp fscache-MDT0000-lwp-MDT0000 fscache-MDT0000-lwp-MDT0000_UUID 5