Netdata/Prometheus

Notes

This page is about connecting Netdata with Prometheus. Netdata collects statistics about the computer, and Prometheus is a time series database.

Information on setting up/installing netdata: Netdata

Prometheus notes: Prometheus

Link to guide: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

Procedure

Install Prometheus

Start by installing Prometheus, following instructions on Prometheus page. Prometheus offers a Linux binary, or you can build from source.

Once you have Prometheus installed, you can run it, and it will start listening on port 9090.

Configure Prometheus

Here is the configuration file used for prometheus (prometheus.yaml in whatever directory the prometheus binary is located):

# Set global configurations:
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: codelab-monitor

# Now configure each endpoint
# that Prometheus should scrape
scrape_configs:

# This is an example endpoint 
# of Prometheus polling its own API
- job_name: prometheus
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9090

# Netdata job: jupiter
- job_name: netdata_jupiter
  params:
    format:
    - prometheus
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /api/v1/allmetrics
  scheme: http
  static_configs:
  - targets:
    - localhost:19999

# Netdata job: (other machines)
# (just change the localhost to the IP of the machine)

Run Prometheus

Note that when you run prometheus, you will need to run it from the same directory as your config file, which should be in the same directory as the prometheus binary.

$ pwd
/opt/prometheus

$ /bin/ls -l
total 70232
drwxr-xr-x 2 charles charles     4096 Jun 12  2017 console_libraries
drwxr-xr-x 2 charles charles     4096 Jun 12  2017 consoles
drwx------ 6 charles charles     4096 Jan 18 17:26 data
-rw-r--r-- 1 charles charles    11357 Jun 12  2017 LICENSE
-rw-r--r-- 1 charles charles     2793 Jun 12  2017 NOTICE
-rwxr-xr-x 1 charles charles 61829345 Jun 12  2017 prometheus
-rw-r--r-- 1 charles charles     1483 Jan 18 14:44 prometheus.yml
-rwxr-xr-x 1 charles charles 10051241 Jun 12  2017 promtool

$ /opt/prometheus/prometheus
INFO[0000] Starting prometheus (version=1.7.1, branch=master, revision=3afb3fffa3a29c3de865e1172fb740442e9d0133)  source="main.go:88"
INFO[0000] Build context (go=go1.8.3, user=root@0aa1b7fc430d, date=20170612-11:44:05)  source="main.go:89"
...

The configuration file is copied from the default, but has a section called "netdata" appended to the end. This creates multiple time series prefixed with "netdata_" from the metrics at the REST API endpoint provided.

Netdata Prometheus REST Endpoint

You'll have to configure Prometheus to tell it about the Netdata REST endpoint that provides data in a Prometheus-friendly format.

Here is what the REST endpoint for Prometheus looks like:

http://192.168.25.236:19999/api/v1/allmetrics?format=prometheus&help=yes

Run Netdata

All you need to do is run Netdata - no backend configuration required.

When you run Netdata, it will automatically provide the REST endpoints without configuration. The model Prometheus uses is for the Prometheus server to query the REST endpoint, rather than waiting for data to be sent to it, so it all works out.

End Result

The end result is, you should see several netdata time series available in Prometheus. You can start typing the name of a time series and a drop-down will show you different time series in the database. You can click "Graph" to visualize the time series, and you can limit the time series to a particular dimension using {dimension="mydimension"}:

Next Steps

Next steps are to spend time getting to know Prometheus. See Prometheus.