[LU-8455] Pacemaker script for Lustre and ZFS

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Assignee: Gabriele Paciucci (Inactive)
• Reporter: Gabriele Paciucci (Inactive)

Description

A new script to be used in Pacemaker to manage ZFS pools and Lustre targets.

This RA is able to manage (import/export) ZFS pools and Lustre Target (mount/umount).

pcs resource create <Resource Name> ocf:heartbeat:LustreZFS \ 
pool="<ZFS Pool Name>" \
volume="<ZFS Volume Name>" \
mountpoint="<Mount Point" \
OCF_CHECK_LEVEL=10

where:

• pool is the pool name of the ZFS resource created in advance
• volume is the volume name created on the ZFS pool during the Lustre format (mkfs.lustre).
• mount point is the mount point created in advance on both the Lustre servers
• OCF_CHECK_LEVEL is optional and enable an extra monitor on the status of the pool

This script should be located in /usr/lib/ocf/resource.d/heartbeat/ of both the Lustre servers with permission 755.

The script provides protection from double imports of the pools. In order to activate this functionality is important to configure the hostid protection in ZFS using the genhostid command.

Default values:

• no defaults

Default timeout:

• start timeout 300s
• stop timeout 300s
• monitor timeout 300s interval 20s

Compatible and tested:

• pacemaker 1.1.13
• corosync 2.3.4
• pcs 0.9.143
• RHEL/CentOS 7.2

Issue Links

• duplicates: LU-8458 Pacemaker script to monitor Lustre servers status

Comments

[Vaughn E. Clinton (Inactive)]

Checked to see what the resource option could locate with respect to ZFS and here's what I got:

pcs resource list | grep -i zfs
ocf:heartbeat:Lustre-MDS-ZFS - Lustre and ZFS management when the MDT and MGT
ocf:heartbeat*:LustreZFS* - Lustre and ZFS management
ocf:llnl:lustre - Lustre ZFS OSD resource agent
ocf:llnl:zpool - ZFS zpool resource agent
ocf:pacemaker:Lustre-MDS-ZFS - Lustre and ZFS management when the MDT and MGT
ocf:pacemaker:LustreZFS - Lustre and ZFS management

ls /usr/lib/ocf/resource.d/pacemaker
ClusterMon Dummy healthLNET HealthSMART LustreZFS pingd Stateful SystemHealth
controld HealthCPU healthLUSTRE Lustre-MDS-ZFS ping remote SysInfo

ls /usr/lib/ocf/resource.d/heartbeat
apache Delay exportfs healthLUSTRE iSCSILogicalUnit LVM nfsnotify oralsnr redis Squid
clvm dhcpd Filesystem iface-vlan iSCSITarget MailTo nfsserver pgsql Route symlink
conntrackd docker galera IPaddr Lustre-MDS-ZFS mysql nginx portblock rsyncd tomcat
CTDB Dummy garbd IPaddr2 LustreZFS nagios ocf-rarun postfix SendArp VirtualDomain
db2 ethmonitor healthLNET IPsrcaddr named oracle rabbitmq-cluster slapd Xinetd

 

ls /usr/lib/ocf/resource.d/llnl/
lustre zpool

The LLNL agents were installed yesterday by another staff member and we were able to successfully create the resources using the LLNL RA scripts but not the Intel ones:

Online: [ mds00 mds01 ]

Full list of resources:

hammer_io6 (stonith:fence_powerman): Started mds00

• hammer_io5 (stonith:fence_powerman): Started mds01*
• lustreMDSPool (ocf::llnl:zpool): Started mds00*
• lustreMGT (ocf::llnl:lustre): Started mds00*
• lustreMDT (ocf::llnl:lustre): Started mds00*

Anyway, if you have any other suggestions, I'd welcome them because I'd prefer using a vendor RA but will settle with the LLNL one for the moment.

Thanks again for the support with this.

Cheers,

[Malcolm Cowe (Inactive)]

From the output, it looks as though PCS cannot find almost any resources. Probably need to check that the packages are installed correctly.

For reference, the packages on my server are:

[root@ct66-mds2 ~]# rpm -qa resource-agents
resource-agents-3.9.5-82.el7_3.6.x86_64
[root@ct66-mds2 ~]# rpm -qa Lustre-ZFS-RA
Lustre-ZFS-RA-0.99.5-1.noarch

The RAs are installed in /usr/lib/ocf/resource.d, in subdirectories for each class. For example, the pacemaker directory on one of my servers looks like this:

[root@ct66-mds2 ~]# ls /usr/lib/ocf/resource.d/pacemaker
ClusterMon  Dummy      healthLNET    HealthSMART     LustreZFS  pingd   Stateful  SystemHealth
controld    HealthCPU  healthLUSTRE  Lustre-MDS-ZFS  ping       remote  SysInfo

The pcs resource list command scans these directories to assemble the list of available RAs. Running pcs resource list with no further arguments should return a large list of available resource agents.

If none of the RAs are showing up, but there are files listed in /usr/lib/ocf/resource.d/{heartbeat,pacemaker}, then it is possible that there is a permissions problem. All the RAs need to have the executable bit set, and on a default install will have mode 755 on all files and directories, owned by root. If they are correct, then perhaps something like SELinux is interfering, although I would hope that that is unlikely.

[Vaughn E. Clinton (Inactive)]

Malcolm,

Thanks for the response!  I really appreciate the help with this since I'm very new at PCS/Pacemaker/Corosync setups.

Anyway, I ran the following command with the syntax as you suggested.  Here's the return from the command:

pcs resource list ocf:pacemaker | awk 'tolower($0) ~ /lustre|lnet/'
Error: No resource agents matching the filter.

I even attempted with heartbeat and here's the return for that attempt:

pcs resource list ocf:heartbeat | awk 'tolower($0) ~ /lustre|lnet/'
Error: No resource agents matching the filter.

I did attempt to create the resources anyway and it failed as with the previous attempts:

pcs resource create hail-mgt ocf:pacemaker:LustreZFS pool="ha-mds" volume="mgt" mountpoint="/lustre/hail/mgmt"

Error: Unable to create resource 'ocf:pacemaker:LustreZFS', it is not installed on this system (use --force to override)

I forgot to add the version of the resource-agents RPM that installed in this environment:

resource-agents-3.9.5-82.el7_3.3.x86_64

**Again, thanks for the assistance

[Malcolm Cowe (Inactive)]

Hi Vaughn,

Try using the path ocf:pacemaker:Lustre-MDS-ZFS, instead of ocf:heartbeat:Lustre-MDS-ZFS. You can also verify the list of available RAs using the command pcs resource list. For example:

[root@ct66-mds2 ~]# pcs resource list ocf:pacemaker | awk 'tolower($0) ~ /lustre|lnet/'
ocf:pacemaker:Lustre-MDS-ZFS - Lustre and ZFS management when the MDT and MGT
ocf:pacemaker:LustreZFS - Lustre and ZFS management
ocf:pacemaker:healthLNET - LNet connectivity
ocf:pacemaker:healthLUSTRE - lustre servers healthy

 

[Vaughn E. Clinton (Inactive)]

I've been trying to use the script to create the HA volumn/dataset resources with the following syntax:

pcs resource create hail-mgt ocf:heartbeat:Lustre-MDS-ZFS pool="ha.mds" volume="mgt" mountpoint="/lustre/hail/mgt"

Each attempt returns the following error:

Error: Unable to create resource 'ocf:heartbeat:Lustre-MDS-ZFS', it is not installed on this system (use --force to override)

I can see that the agent script, Lustre-MDS-ZFS, is dropped into the correct location when I run this syntax with the debug option enabled.  I also see the script being ran with a return 0 value. I'm not exactly sure what the problem is. Could it be missing some binary that I'm not seeing in the debug output? Anyway, I would greatly appreciate some guidance with solving this.

Here are the details about my configuration:

Red Hat Enterprise Linux Server release 7.3 (Maipo)
pcs-0.9.152-10.el7.x86_64
pacemaker-1.1.15-11.el7_3.2.x86_64
corosync-2.4.0-4.el7.x86_64

fence-agents-common-4.0.11-47.el7_3.2.x86_64
fence-agents-powerman-4.0.11-7.ch6.x86_64
libxshmfence-1.2-1.el7.x86_64

This is being deployed in a diskless 2 node HA Lustre environment. Please let me know if you require me to open a ticket concerning this issue.

[Gabriele Paciucci (Inactive)]

Added a new version 0.99.5 that fix the amount issue

[Andreas Dilger]

Using "umount -f" will evict all of the clients at unmount time, which means they will get IO errors for any in-flight writes when the server is started again. This is useful if the administrator knows that the clients have already been stopped, for example.

I don't think that using the "-f" flag will significantly reduce the amount of time that the unmount will take, but if you have numbers to that effect it would be interesting to see. It may be possible to improve the performance of the non-f unmount if that is a real issue.

[Gabriele Paciucci (Inactive)]

Hi bmerchant thank you for this feedback. I would like to consult with adilger about this.
I'm using umount -f because I don't want wait long time to close the file system if several clients are connected. If I hit the stop timeout in pacemaker, pacemaker will perform a stonith of the node.
But at the same time the -f option is disabling the recovery and this could cause issues with data?
adilger could you advice please. Thank you.

[Bradley Merchant (Inactive)]

We have been using this script and ran into an issue on manual failover (i.e. pcs resource disable/enable or move/relocate). On resource stop the script force unmounts Lustre targets (umount -f). In live tests this caused the recovery flags to not be set on OSTs, causing client eviction on remount. Removing the -f flag resolves the problem, but we are curious if removing it could present any other issues.

I do notice Christopher Morrone's script omits the -f flag.

[Andreas Dilger]

There is a considerable amount of duplication between this ticket and LU-8458. One of them should probably be closed.

[Gabriele Paciucci (Inactive)]

Hi morrone,
as I already mentioned in our discussion, I found problems (and weakness) in pacemaker to manage two resources colocated. One of the problem was to move (from your example) MGS from jet1 to jet2. The only way to do this is moving jet1-1, but this means that if the MGS resource failed, the other resource is not moved... I tried also using the resource group concept but without success.
I tested this in RHEL 6 maybe in RHEL 7 they fixed, this issue. Could you please test this?

I also suggest to use the resource stickness option to avoid any failback.