2018/January

Task list for January:

in progress

data engineering

Data engineering scenarios

• workflow with large amounts of data
• role of logging and log files
• role of dashboards
• Google Cloud/Review

2018/January/Data Engineering

kali current and future

Live USB problems:

• Not persistent with preferences (keyboard)
• Logging, code, scripts onboard.
• Kali/Persistent USB

Bro:

• I'm not sure how this thing works. I just know it logs everything.
• How does it run, where do I configure it?
• What should my network architecture look like?
• Do I need a network tap?

Network monitoring ISO files:

• Security Onion
• SELKS consisting of Suricata  · Elasticsearch  · Logstash  · Kibana  · Scirius  · EveBox
• Rock Network Security Monitor
• Exactly what I'm trying to do: [1] (except, with SELKS)

Next months:

• Networks: MITM, wifi, evil ap, honeypot
• payloads: exploits, payload gen, remote/reverse shell, metasploit
• pw credentials wall of sheep in a box

esp 8266

sensor + wifi, data sent via json

can see how high speed data acquisition could get tricky

microphone? usb?

blog posts

• Blog posts:
  • Knuth permutation generation
  • 4x4 Rubik's Cube: Part 1: Representations
  • 4x4 Rubik's Cube: Part 2: Permutations
  • 4x4 Rubik's Cube: Part 3: Cycles
  • 4x4 Rubik's Cube: Part 4: Orders
  • Project Euler 172
  • Project Euler 502
  • Bots - RMM Queueau Poem
  • Google Data Engineering Certification blog post and notes highlights

genealogy

Genealogy photos:

• Photos cropped/organized by family
  • 2011
  • 2017
  • Rename scheme
  • Notes - A2k11
  • Notes - R2k11
  • Notes - A2k17
  • Notes - K2k17
  • Notes - R2k17
• Send email to fam with link on Dropbox

Writing:

• Pauline and Bruce chapters
• Historical research planning

notes

• Shore up notes:
  • Experiment design: https://charlesreid1.com:3000/notes/experiment-design
  • Linear models: https://charlesreid1.com:3000/notes/linear-models
  • Rubiks cube: https://charlesreid1.com:3000/notes/rubiks-notes

2018/January/Notes Repositories

complete

project euler

• Project Euler:
  • Project Euler: Problem 172
  • Clean up problems 52-67

Project Euler/172

rubiks cube

• Rubiks Cube:
  • Calculating the order of a permutation (see https://math.stackexchange.com/questions/332146/efficient-method-to-determine-the-order-of-a-permutation-in-s-n)

2018/January/Rubiks Cube

• Rubiks Cube/Numbers
• Rubiks Cube/Tuple
• Rubiks Cube/Permutations

desk reorg

Reorganize and rearrange desk

• Move book boxes to room, move table from room to main area
• Consolidate plastic tubs
• Computer lab - 2 laptops + desktop
• Switch to dropbear as the main router

master git repos

Master repo layout:

• Create master repo for each organization
• Each repo in org forms submodule
• Utility scripts to update to latest, and to commit latest

bots

• Tripos bot
  • Work out small page latex
  • Get questions into latex format
  • Script to compile latex to pdf, Imagemagick latex to jpg
  • Create Rainbow Mind Machine bot
  • Create Twitter account
  • Authenticate
  • Set up and run on Rojo
  • Debug - it's just not tweeting.

• Apollo Bot
  • Fix lengths of dialogue per speaker
  • Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
  • Get back online

• Milton Bot
  • Seems okay... but still having some permission problem
  • Problem was, could not access keys, b/c not run from correct directory.
  • Need to fix run_bots_in_screen.sh
  • Get back online

• Ginsberg Bot
  • Seems okay, but permission problem (see above)
  • Get back online

Tripos Bot

dotfiles

dotfiles:

• jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
• lite/vanilla/docker

dotfile submodules/organization:

• one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
• organization over convenience - not as much sharing because generally static/fixed

debian deployment:

• rojo copy dotfile changes over to debian repo
• rojo apply changes, ensure nothing broken
• jupiter apply changes, ensure nothing broken

mac deployment:

• copy dotfiles from debian
• copy python from debian
• apply changes, ensure nothing broken

install scripts:

• debian apt-get install script Jupiter
• mac brew install Homebrew/Post Install
• kali apt-get install Kali/Post Install  · Kali Raspberry Pi/Post Install
• rpi apt-get install RaspberryPi/Post Install

Dotfiles

• https://charlesreid1.com:3000/dotfiles/vanilla
• https://charlesreid1.com:3000/dotfiles/dotfiles-master
• https://charlesreid1.com:3000/dotfiles/rpi
• https://charlesreid1.com:3000/dotfiles/mac
• https://charlesreid1.com:3000/dotfiles/kali
• https://charlesreid1.com:3000/dotfiles/debian

kali deployment and rpi deployment to be fixed later

kali

Jupiter:

• 2 TB hard drive (ordered)
• Debian (100 gb windows)
• Kali via persistent live usb configured for Jupiter

VPN:

• Charlesreid1 connection
• Anonymous layer PIA on dropbear
• Anonymous browsing notes, Privoxy, &c.
• Tinc VPN

Instrumentation:

• netdata
• network traffic

Networking tools:

• Kali Tools installation and workflow
• Bro installation

netdata

Netdata setup:

• Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
• Netdata running on each, and Netdata instances accessible from on the network
• Install Prometheus on jupiter
• Connect Netdata on basilisk and morpheus to Prometheus on jupiter

VPN mesh network:

• Tinc
• Working okay on the home network.

Netdata security:

• Set up vpn via tinc on rojo
• Set up netdata on rojo
• Bind netdata to vpn ip address only
• Verify rojo netdata instance protected

Digital ocean guide - what role does nginx play?

Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.

Netdata/Security

Prometheus/Security

Charlesreid1.com data

• Share netdata securely
• VPN connection to server
• Logs, network traffic, loading, pages, processes

Netdata

Prometheus