2018/March

overarching goal

2018/Data Project

Current stage: spy project (preparing Dockerfiles, testing additional components before yeti deployment)

aws things

AWS writeup for DIB lab group: AWS Writeup

Amazon VPC and VPN

• set up private test node with vpc
• if no internet gateway, how can we update? if have internet gateway, can anyone accses necessarily?
• networking patterns - private and public subnet, single public subnet, one gateway with internet for mult. subnets
• use this as a testbed for banana pi hardware

monitoring and dashboards

netdata

• adding python metrics
• workflow? need to restart?
• accessing via api - nomenclature/special considerations
• file size, download progress, etc. from python

prometheus

• how to structure data directory
• on the 1st of the month:
• find any data older than 1 month
• archive it and put it in a bucket
• purge data from system

grafana

• building more complex dashboards
• sharing dashboards

dashboards

current status: resolved to use mongodb

next step goals:

• run mongodb query in javascript
• collect more data
• visualize data for single chart/single time series with D3
• visualize more data using Grafana

toy problem:

• use instrumented traveling salesman problem (or rubiks cube code, or some other project euler code)
• proof of concept use of monitoring
• greenfield deployment of netdata

goal:

• understand/monitor/understand large complex systems
• minimize time to set up database, add metrics, visualize, gain insight, repeat

toy problem:

• instrumented TSP or other

toy problem goal

instrument a code, instrument a node, combine collected data about both

• pick a computationally intensive problem
• pick a node platform
• collect the data - mongodb

instrument a TSP problem code, show a proof of concept of what we want to be able to record/measure/visualize

focus on greenfield deployments - install netdata, collect stats, visualize them

d3 viz

10 visualizations:

• wiki edits calendar visualization https://charlesreid1.com/wiki_calendar and D3/Calendar
• git calendar visualization https://charlesreid1.com/git_calendar and D3/Calendar
• game of life https://charlesreid1.com/life and D3/Life
• map series: Javascript/Maps
  • leaflet maps with click behavior
  • leaflet maps linked to c3/d3 charts
  • twin linked leaflet maps

more calendars:

• wikipedia article edit calendars
• dates... color dimension, pref. binary...

Fix a shrubbery

Also see D3x10

blog posts

• Project Euler/502
• D3/Life visualization
• Bots
• bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap
• aws

bots

Still chuggin with the charlesreid1 spreadsheet. Slogging through the litany of wiki pages. Google Sheets. https://charlesreid1.com:3000/charlesreid1/charlesreid1-awesome

With the deployment of charlesreid1 bot, we want to take the time to do it more thoroughly:

• bot instrumentation
• dashboard
• monitoring bot status
• statistics
• display latest tweet
• bot dashboard built with grafana

Bots/Charlesreid1

Bots/Instrumentation

apollo bots:

• 14/15/16/17
• incorporate lunar surface dialogue

apollo references:

• link to apollo 11 audio: https://www.nasa.gov/mission_pages/apollo/40th/a11_audio_db.html
• Spacelog (mercury, gemini, pre-11 apollo missions): http://spacelog.org/
• apollo surface journals: https://history.nasa.gov/alsj/main.html

Category:Bots

Bots/New Apollo

genealogy

Genealogy photos:

• Photos cropped/organized by family
  • 2011
  • 2017
  • Rename scheme
  • Notes - A2k11
  • Notes - R2k11
  • Notes - A2k17
  • Notes - K2k17
  • Notes - R2k17
• Send email to fam with link on Dropbox

Writing:

• Pauline and Bruce chapters
• Historical research planning

stretch goal

test backup and restore scripts

automate automate automate

• use an AWS node or a GC node to actually test the backups, start to finish.
• creating a second node that's a complete and total backup of the first.
• in the process of doing this, we're going to have a lot of questions to answer and software to fix.
• letsencrypt certificate generation
• ssh

We are one step toward that goal with scripts that are automating site updates, committing to git repos, pulling latest versions of data, etc.

Now that we are familiar with subprocess for non-trivial tasks, we can move forward with scripts to automate testing.

back burner

See 2018/February#back burner