2018/Data Project: Difference between revisions
From charlesreid1
| Line 15: | Line 15: | ||
==Phase 2: MongoDB and MongoExpress== | ==Phase 2: MongoDB and MongoExpress== | ||
We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers | We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki. | ||
* '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc. | * '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc. | ||
* '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity. | * '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity. | ||
| Line 24: | Line 24: | ||
* Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress | * Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress | ||
* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data | * Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data | ||
==Phase 3: Graphite and Grafana== | |||
Next, we deployed a [[Graphite]] container to hold time series and a [[Grafana]] container to create dashboards from it. | Next, we deployed a [[Graphite]] container to hold time series and a [[Grafana]] container to create dashboards from it. | ||
Revision as of 20:41, 10 February 2018
Project Overview
The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.
Phase 1: Netdata and Prometheus
First, we set up Netdata to dump to a Prometheus database.
- Pros: Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
- Cons: Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
- Netdata
- Prometheus
Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.
Phase 2: MongoDB and MongoExpress
We then set up MongoDB and MongoExpress in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki.
- Pros: MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
- Cons: No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.
- MongoDB
- MongoExpress
- Pywikibot
- Link to MongoDB docker files: https://charlesreid1.com:3000/docker/d-mongodb
- Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
- Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
Phase 3: Graphite and Grafana
Next, we deployed a Graphite container to hold time series and a Grafana container to create dashboards from it.
- Pros: Containerized solution, like MongoDB
- Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite
- Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-graphite