2018/Data Project
From charlesreid1
Project Overview
The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.
Phase 1: Netdata and Prometheus
First, we set up Netdata to dump to a Prometheus database.
- Pros: Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
- Cons: Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
- Netdata
- Prometheus
Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.
Phase 2: MongoDB and MongoExpress
We then set up MongoDB and MongoExpress in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki.
- Pros: MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
- Cons: No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.
- MongoDB
- MongoExpress
- Pywikibot
- Link to MongoDB docker files: https://charlesreid1.com:3000/docker/d-mongodb
- Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
- Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
Phase 3: Graphite and Grafana
Next, we deployed a Graphite container to hold time series and a Grafana container to create dashboards from it.
- Pros: Containerized solution, like MongoDB
- Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite
- Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-graphite