Prometheus
From charlesreid1
Prometheus is a time series database backend. It connects to data sources like Netdata and graph libraries like Grafana, making Prometheus a solid choice for a time series database.
Mostly followed this guide from Netdata wiki on how to set up Netdata with Prometheus: [1]
Installing
The following installation procedure will download the pre-built Linux binary and put it in /opt/prometheus:
curl -L 'https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gz' \
-o /tmp/prometheus.tar.gz
mkdir /opt/prometheus
tar -xf /tmp/prometheus.tar.gz -C /opt/prometheus/ --strip-components 1
Using
To run, just point your browser to localhost:9090 (or, if you are on another machine, access port 9090 of the machine running Prometheus.)
Word of warning, the Prometheus web portal is insecure by default.
Queries
To run a query, start typing the name of a time series and you will see an autocomplete drop-down menu from Prometheus.
You can type the name of a time series and press enter (and click the Graph button) to see a graph of your time series quantity.
However, as you collect and group more metrics together, you can also filter and match different time series.
Link: Querying Basics: https://prometheus.io/docs/prometheus/latest/querying/basics/
Dimensions
As an example of dimensions: we are collecting data with Netdata and sending it to Prometheus. Netdata collects several pieces of information about the CPU utilization under the netdata_cpu_cpu_percentage_average variable. Prometheus collects these as a single time series with multiple dimensions. So we can specify the "dimension" keyword to filter on dimensions. The dimensions are things like "user", "system", "idle", etc.
To graph all dimensions of this time series, type this in the query box:
netdata_cpu_cpu_percentage_average
To graph only the system CPU utilization, specify the "system" dimension:
netdata_cpu_cpu_percentage_average{dimension="system"}
Jobs
If we end up monitoring several services on a single machine, or if we monitor multiple machines across a network, we will have different sections of the Prometheus configuration file, corresponding to different jobs. For example, I might have three netdata instances running on three computers: jupiter, basilisk, and morpheus.
When we query data in Prometheus, we can specify which job's data to use:
netdata_cpu_cpu_percentage_average{dimension="system",job="netdata_jupiter"}
Alternatively, we can match multiple jobs by using regular expressions and the =~ operator:
netdata_cpu_cpu_percentage_average{dimension="system",job=~"netdata_.*"}
and the output:
Grafana
http://docs.grafana.org/installation/debian/
https://blog.hda.me/2017/01/09/using-netdata-with-influxdb-backend.html