2018/February
From charlesreid1
overarching goal
dashboards
current status: resolved to use mongodb
next step goals:
- run mongodb query in javascript
- collect more data
- visualize data for single chart/single time series with D3
- visualize more data using Grafana
toy problem:
- use instrumented traveling salesman problem (or rubiks cube code, or some other project euler code)
- proof of concept use of monitoring
- greenfield deployment of netdata
goal:
- understand/monitor/understand large complex systems
- minimize time to set up database, add metrics, visualize, gain insight, repeat
toy problem:
- instrumented TSP or other
toy problem goal
instrument a code, instrument a node, combine collected data about both
- pick a computationally intensive problem
- pick a node platform
- collect the data - mongodb
instrument a TSP problem code, show a proof of concept of what we want to be able to record/measure/visualize
focus on greenfield deployments - install netdata, collect stats, visualize them
d3 viz
10 visualizations:
wiki edits calendar visualizationhttps://charlesreid1.com/wiki_calendar and D3/Calendargit calendar visualizationhttps://charlesreid1.com/git_calendar and D3/Calendargame of lifehttps://charlesreid1.com/life and D3/Life- map series: Javascript/Maps
- leaflet maps with click behavior
- leaflet maps linked to c3/d3 charts
- twin linked leaflet maps
more calendars:
- wikipedia article edit calendars
- hourly calendar?
back burner:
- time series for code profiling D3/Time Series
- brushable time series chart D3/Brushable Time Series
- mongodb-javascript connection
- bots or scraped data daily viz
Fix a shrubbery
Also see D3x10
blog posts
Project Euler/172D3/Calendar visualization- Project Euler/502
- D3/Life visualization
- Bots
- bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap
bots
charlesreid1 bot:
- build spreadsheet, parse to build awesome list and bot
- https://git.charlesreid1.com/charlesreid1/charlesreid1-awesome
bot instrumentation:
- dashboard, monitoring, statistics, status
- bot dashboard with grafana
apollo bots:
- 14/15/16/17
- incorporate lunar surface dialogue
apollo references:
- link to apollo 11 audio: https://www.nasa.gov/mission_pages/apollo/40th/a11_audio_db.html
- Spacelog (mercury, gemini, pre-11 apollo missions): http://spacelog.org/
- apollo surface journals: https://history.nasa.gov/alsj/main.html
genealogy
Genealogy photos:
- Photos cropped/organized by family
20112017Rename schemeNotes - A2k11- Notes - R2k11
- Notes - A2k17
- Notes - K2k17
- Notes - R2k17
- Send email to fam with link on Dropbox
Writing:
- Pauline and Bruce chapters
- Historical research planning
stretch goal
test backup and restore scripts
automate automate automate
- use an AWS node or a GC node to actually test the backups, start to finish.
- creating a second node that's a complete and total backup of the first.
- in the process of doing this, we're going to have a lot of questions to answer and software to fix.
- letsencrypt certificate generation.
back burner
networking
monitoring hardware
network tap/switch- wired router
new router:
- website with database of embedded dev boards: board-db.org
- Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
- Banana Pi R2 Link: [1]
data streams
data streams:
- sensor data from a physical sensor (raspberry pi, gpio, radio, sdr)
- rojo/jupiter log data
- network log info from bro
- twitter/news scraping
bokeh viz
bokeh: https://github.com/bokeh/bokeh
interactive dashboards:
- glue between command-line scripts and visual graphs
- don't worry about "live" ajax refreshing - just focus on analytics and visualization
complete
data store (db)
the database is the central thread for everything.
get a database solution up and running over the management lan.
minimize friction and time to bring up/explore/check new collection.
Note: minimizing friction mainly just comes down to (a) getting it running, thank you very much docker, and (b) familiarity with syntax. everything else is pretty seamless.
completed data streams
completed data streams:
wiki data - historical evolution, graph analysishttps://git.charlesreid1.com/wiki/charlesreid1-wiki-datarojo/jupiter/maya/cronus hardware status - collectd, netdataGraphite · https://git.charlesreid1.com/docker/d-graphite
We're not using Collectd anymore, we're using Netdata
Docker/System Stats is a possible solution to collectd
wiki visualization
visualizations:
- calendar of edits
calendar of character counts of edits
calendar visualization: https://charlesreid1.com/calendar
git
More organizing and creating git repositories:
- data organization for repos dealing with data collection, monitoring, or storing raw data
- created repos for raw data - census, maps, git, wiki
- these are contained in a data-master repo that is cloned directly into charlesreid1.com htdocs dir
data is accessible at charlesreid1.com/data
d3 on git.charlesreid1.com:
- census data for maps: https://git.charlesreid1.com/charlesreid1/census-data
- d3 maps: https://git.charlesreid1.com/d3/maps
d3 calendar: https://git.charlesreid1.com/d3/charlesreid1-calendar
data scripts
dotfiles/debian/{rojo_scripts,jupiter_scripts}:
rojo git script:
- set up script to pull latest git commit data
- make csv with commit data
- push new csv to git data repo
- tested
crontab
jupiter wiki script:
- set up script to compile latest wiki edit/graph data
- make csv/json with data
- push new data to wiki data repo
- tested
crontab