2018/Data Project
From charlesreid1
Project Overview
The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.
Phase 1: Netdata and Prometheus
First, we set up Netdata to dump to a Prometheus database.
- Pros: Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
- Cons: Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
- Netdata
- Prometheus
Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.
Phase 2: MongoDB and MongoExpress
We then set up MongoDB and MongoExpress in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki.
- Pros: MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
- Cons: No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.
- MongoDB
- MongoExpress
- Pywikibot
- Link to MongoDB docker files: https://charlesreid1.com:3000/docker/d-mongodb
- Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
- Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
Phase 2b: Collectd
We struggled a LOT with Collectd, mainly because we wanted to use the collectd plugin to write to MongoDB. Unfortunately, this was the only plugin that seemed impossible to install.
See Collectd page.
(This is all installation stupidity. I tried installing collectd with aptitude, no plugins. Then the core, no plugins. Then installing from source, and MongoDB plugin did not work. Struggling to get collectd to link to MongoDB. Needed custom config or something. Then I just gave up, and re-installed collectd core, and the library was there, but it was complaining it couldn't find it. In the end, I totally abandoned the attempt to get collectd to talk to mongodb. Could probably use a collectd docker and fix this whole issue.)
Phase 3: Graphite and Grafana
Next, we deployed a Graphite container to hold time series and a Grafana container to create dashboards from it.
- Pros: Containerized solution, like MongoDB
- Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite
- Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-graphite