2018/February: Difference between revisions

From charlesreid1
Line 3: Line 3:
[[2018/Data Project]]
[[2018/Data Project]]


===data streams===
===database and monitoring===


data streams:
rojo:
* sensor data from a physical sensor (raspberry pi, gpio, radio, sdr)
* set up script to pull latest wiki edit data
* rojo/jupiter log data
* subprocess
* network log info from bro
* pull update theme
* twitter/news scraping
* pull update charlesreid1.com
* pelican content
* copy to htdocs (use rsync)


https://community.rackspace.com/products/f/public-cloud-forum/6800/how-to-set-up-monitoring-stack-using-collectd-graphite-grafana-and-seyren-on-ubuntu-14-04
jupiter:
* set up script to regenerate wiki edit data
* subprocess
* push update charlesreid1.com


===d3 viz===
===d3 viz===
Line 23: Line 28:
* mongodb-javascript connection
* mongodb-javascript connection
* bots or scraped data daily viz
* bots or scraped data daily viz
===bokeh viz===
bokeh: https://github.com/bokeh/bokeh
interactive dashboards:
* glue between command-line scripts and visual graphs
* don't worry about "live" ajax refreshing - just focus on analytics and visualization


===dashboards===
===dashboards===


current status of dashboards:
current status of dashboards:
* collecting data from data sources mentioned above
* using mongodb as principal platform
* collecting data
* visualizing data for single charts/single time series with d3
* visualizing data for single charts/single time series with d3
* building more complex dashboards with [[Grafana]]
* building more complex dashboards with [[Grafana]]
Line 43: Line 41:
* minimize time to set up database, add metrics, visualize, gain insight, repeat
* minimize time to set up database, add metrics, visualize, gain insight, repeat


===database and monitoring===
toy problem:
 
* instrumented TSP or other
rojo:
* set up script to pull latest wiki edit data
* subprocess
* pull update theme
* pull update charlesreid1.com
* pelican content
* copy to htdocs (use rsync)
 
jupiter:
* set up script to regenerate wiki edit data
* subprocess
* push update charlesreid1.com
 
===networking===
 
monitoring hardware
* <s>network tap/switch</s>
* wired router
 
new router:
* website with database of embedded dev boards: [https://www.board-db.org/search.php?q=&ram_min=&cpu_speed_min=&cpu_cores=&cpu_arch=any&price_min=&price_max=&storage_min=&gpio_min=&usb_min=&type=&lan=on&multiple_lan=on&lan_speed=0&order=price&order_d=a#results board-db.org]
* Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
* Banana Pi R2 Link: [https://www.aliexpress.com/item/BPI-R2-Banana-PI-R2-Smart-Open-source-Wireless-Router-BPI-R2-Smart-Home-Control-Device/32825598599.html]


===blog posts===
===blog posts===
Line 85: Line 60:


[[Bots/Charlesreid1]]
[[Bots/Charlesreid1]]


bot instrumentation:
bot instrumentation:
Line 93: Line 66:


[[Bots/Instrumentation]]
[[Bots/Instrumentation]]


apollo bots:
apollo bots:
Line 125: Line 97:
* Pauline and Bruce chapters
* Pauline and Bruce chapters
* Historical research planning
* Historical research planning
==back burner==
===networking===
monitoring hardware
* <s>network tap/switch</s>
* wired router
new router:
* website with database of embedded dev boards: [https://www.board-db.org/search.php?q=&ram_min=&cpu_speed_min=&cpu_cores=&cpu_arch=any&price_min=&price_max=&storage_min=&gpio_min=&usb_min=&type=&lan=on&multiple_lan=on&lan_speed=0&order=price&order_d=a#results board-db.org]
* Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
* Banana Pi R2 Link: [https://www.aliexpress.com/item/BPI-R2-Banana-PI-R2-Smart-Open-source-Wireless-Router-BPI-R2-Smart-Home-Control-Device/32825598599.html]
===data streams===
data streams:
* sensor data from a physical sensor (raspberry pi, gpio, radio, sdr)
* rojo/jupiter log data
* network log info from bro
* twitter/news scraping
https://community.rackspace.com/products/f/public-cloud-forum/6800/how-to-set-up-monitoring-stack-using-collectd-graphite-grafana-and-seyren-on-ubuntu-14-04
===bokeh viz===
bokeh: https://github.com/bokeh/bokeh
interactive dashboards:
* glue between command-line scripts and visual graphs
* don't worry about "live" ajax refreshing - just focus on analytics and visualization


==complete==
==complete==

Revision as of 00:38, 16 February 2018

overarching goal

2018/Data Project

database and monitoring

rojo:

  • set up script to pull latest wiki edit data
  • subprocess
  • pull update theme
  • pull update charlesreid1.com
  • pelican content
  • copy to htdocs (use rsync)

jupiter:

  • set up script to regenerate wiki edit data
  • subprocess
  • push update charlesreid1.com

d3 viz

10 visualizations:

dashboards

current status of dashboards:

  • using mongodb as principal platform
  • collecting data
  • visualizing data for single charts/single time series with d3
  • building more complex dashboards with Grafana

goal:

  • understand/monitor/understand large complex systems
  • minimize time to set up database, add metrics, visualize, gain insight, repeat

toy problem:

  • instrumented TSP or other

blog posts

bots

charlesreid1 bot:

Bots/Charlesreid1

bot instrumentation:

  • dashboard, monitoring, statistics, status
  • bot dashboard with grafana

Bots/Instrumentation

apollo bots:

  • 14/15/16/17
  • incorporate lunar surface dialogue

apollo references:

Category:Bots

Bots/New Apollo

genealogy

Genealogy photos:

  • Photos cropped/organized by family
    • 2011
    • 2017
    • Rename scheme
    • Notes - A2k11
    • Notes - R2k11
    • Notes - A2k17
    • Notes - K2k17
    • Notes - R2k17
  • Send email to fam with link on Dropbox

Writing:

  • Pauline and Bruce chapters
  • Historical research planning

back burner

networking

monitoring hardware

  • network tap/switch
  • wired router

new router:

  • website with database of embedded dev boards: board-db.org
  • Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
  • Banana Pi R2 Link: [1]

data streams

data streams:

  • sensor data from a physical sensor (raspberry pi, gpio, radio, sdr)
  • rojo/jupiter log data
  • network log info from bro
  • twitter/news scraping

https://community.rackspace.com/products/f/public-cloud-forum/6800/how-to-set-up-monitoring-stack-using-collectd-graphite-grafana-and-seyren-on-ubuntu-14-04

bokeh viz

bokeh: https://github.com/bokeh/bokeh

interactive dashboards:

  • glue between command-line scripts and visual graphs
  • don't worry about "live" ajax refreshing - just focus on analytics and visualization

complete

data store (db)

the database is the central thread for everything.

get a database solution up and running over the management lan.

minimize friction and time to bring up/explore/check new collection.

Note: minimizing friction mainly just comes down to (a) getting it running, thank you very much docker, and (b) familiarity with syntax. everything else is pretty seamless.

completed data streams

completed data streams:

We're not using Collectd anymore, we're using Netdata

Docker/System Stats is a possible solution to collectd

wiki visualization

visualizations:

  • calendar of edits
  • calendar of character counts of edits

calendar visualization: https://charlesreid1.com/calendar

flags

Retrieved from "https://charlesreid1.com/w/index.php?title=2018/February&oldid=24270"