2018/February

overarching goal

2018/Data Project

data streams

data streams:

• sensor data from a physical sensor (raspberry pi, gpio, radio, sdr)
• rojo/jupiter log data
• network log info from bro
• twitter/news scraping

https://community.rackspace.com/products/f/public-cloud-forum/6800/how-to-set-up-monitoring-stack-using-collectd-graphite-grafana-and-seyren-on-ubuntu-14-04

d3 viz

10 visualizations:

• wiki edits calendar visualization - D3/Calendar
• game of life - Javascript/Game of Life
• force graph chart - D3/Force Graph
• git calendar visualization
• brushable time series charts D3/Time Series
• tweets, scraped data, daily scripts visualization - data checkin script to charlesreid1-src

bokeh viz

bokeh: https://github.com/bokeh/bokeh

interactive dashboards:

• glue between command-line scripts and visual graphs
• don't worry about "live" ajax refreshing - just focus on analytics and visualization

dashboards

monitoring with dashboards:

• various data sources mentioned above
• building visualizations and graph dashboards with Grafana
• building and designing monitoring systems for understanding complex, live, evolving systems
• minimizing time to setup, add metrics, gain insight, visualize, repeat

dashbaord software:

• Grafana
• Graphite

database and monitoring

rojo:

• set up script to pull latest wiki edit data
• subprocess
• pull update theme
• pull update charlesreid1.com
• pelican content
• copy to htdocs (use rsync)

jupiter:

• set up script to regenerate wiki edit data
• subprocess
• push update charlesreid1.com

networking

monitoring hardware

• network tap/switch
• wired router

new router:

• website with database of embedded dev boards: board-db.org
• Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
• Banana Pi R2 Link: [1]

blog posts

• Project Euler/172
• Project Euler/502
• Bots
• bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap
• d3 calendar visualization

bots

charlesreid1 bot:

• build spreadsheet, parse to build awesome list and bot
• https://charlesreid1.com:3000/charlesreid1/charlesreid1-awesome

Bots/Charlesreid1

bot instrumentation:

• dashboard, monitoring, statistics, status
• bot dashboard with grafana

Bots/Instrumentation

apollo bots:

• 14/15/16/17
• incorporate lunar surface dialogue

apollo references:

• link to apollo 11 audio: https://www.nasa.gov/mission_pages/apollo/40th/a11_audio_db.html
• Spacelog (mercury, gemini, pre-11 apollo missions): http://spacelog.org/
• apollo surface journals: https://history.nasa.gov/alsj/main.html

Category:Bots

Bots/New Apollo

genealogy

Genealogy photos:

• Photos cropped/organized by family
  • 2011
  • 2017
  • Rename scheme
  • Notes - A2k11
  • Notes - R2k11
  • Notes - A2k17
  • Notes - K2k17
  • Notes - R2k17
• Send email to fam with link on Dropbox

Writing:

• Pauline and Bruce chapters
• Historical research planning

complete

data store (db)

the database is the central thread for everything.

get a database solution up and running over the management lan.

minimize friction and time to bring up/explore/check new collection.

Note: minimizing friction mainly just comes down to (a) getting it running, thank you very much docker, and (b) familiarity with syntax. everything else is pretty seamless.

completed data streams

completed data streams:

• wiki data - historical evolution, graph analysis https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
• rojo/jupiter/maya/cronus hardware status - collectd, netdata Graphite  · https://charlesreid1.com:3000/docker/d-graphite

We're not using Collectd anymore, we're using Netdata

Docker/System Stats is a possible solution to collectd

wiki visualization

visualizations:

• calendar of edits
• calendar of character counts of edits

calendar visualization: https://charlesreid1.com/calendar