[LU-11438] Collectd plugin for collecting Lustre statistics

• Status: Closed
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Li Xi
• Assignee: Li Xi
• Resolution: Won't Fix

Description

We implemented plugins of Collectd to collect the statistics from /proc or /sys entries of Lustre. The plugin as well as Collectd itself has been proven to be lightweighted, efficient and scalable. This collector could be used for a lot kinds of monitoring systems, mostly due to the good capability and features of Collectd. Any one can use that plugin to collect information and send the metrics into monitoring system and databases including Ganglia, OpenTSDB, Influxdb, Graphite, Zabbix, etc.

Several  plugins are added to Collectd of us to support various functions.

• Filedata plugin: The Filedata plugin is able to collect data by reading and parsing a set of files. An XML-formatted definition file is required for the Filedata plugin to understand which files to read and how to parse these files. The most common usage of the Filedata plugin is to collect metrics through /proc interfaces of a running Lustre system.
• Ganglia plugin: The Ganglia plugin can send metrics collected by a collectd client daemon to Ganglia server.****
• SSH plugin: The SSH plugin is able to collect metrics by running commands on remote hosts by using SSH connections. The SSH plugin is used to collect metrics from DDN SFA Storage. Like the GPFS plugin and the IME plugin, the SSH plugin shares the similar definition file format and configuration format with the Filedata
• Stress plugin: The Stress plugin can push a large amount of metrics to server from collectd client in order to benchmark the performance of the collecting system under high pressure.
• Stress2 plugin: Enhanced version of Stress plugin. The format of pushed metrics can be flexibly configured to simulate different real metrics.
• Zabbix plugin: The Zabbix plugin is used to send metrics from collectd to Zabbix system.

Merging these plugins into the master branch of Collectd would be greate, but need a lot of work. Including:

1. Rebase it to latest master branch.
2. Add XML definition files into the repository. It would be a question whether to matain a XML file in the Lustre repository to enable Collectd to know the exact the metric layout of Lustre /proc files.
3. Example of Collectd.conf

Following is the project:

https://github.com/DDNStorage/collectd