From charlesreid1

Task list for January:

in progress

data engineering

Data engineering scenarios

  • workflow with large amounts of data
  • collectd on widy and others
  • logging and log files
  • dashboards
  • Google Cloud/Review

network integration:

  • everything is related...
  • elasticsearch can search json, provides searchable interface to data
  • log data into json format (mongo)
  • collectd system metrics into json format (mongo)
  • bot monitoring metrics, other metrics.... all into json format (mongo)
  • then we just need to figure out the mongodb - grafana hookups

Master plan:

  • the master plan is emerging
  • elasticsearch link: [1]

2018/January/Data Engineering

kali networking

(on hold - tap hardware)

Network monitoring configuration setup:


  • isp-provided router (internet-enabled port)
  • network tap device
  • network monitoring server on a separate management lan
  • "safe" gateway router for everybody

Set up the tap:

  • port 1 of the tap connects to isp
  • port 2 of the tap connects to safe router gateway, local wifi router
  • port 3 of the tap connects to the network monitor, which is receiving network traffic capture from the tap device


  • Logs everything - but how to configure it? how to run it? what is the big picture?
  • How does it fit into the network architecture - is it analyzing the pile of packets that's coming in to the security onion server?

Network monitoring distros:

What this enables:

  • hardware level interception point
  • arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point

Next steps:

  • Networks: MITM, wifi, evil ap, honeypot
  • metasploit, sslstrip, certificate, frankencert
  • focus on communications and networking
  • pw credentials wall of sheep in a box

charlesreid1 wiki

scraping page links and history into database


fixing dropbear

dropbear un-bricking:

  • open the case
  • to unbrick the router, we need to install stuff on the raspberry pi.
  • to install stuff on the raspberry pi, we need to reflash the sd card.
  • to reflash the sd card, we need to figure out which image to flash with. (used kali.)
  • to get pi to join wifi, need to plug into physical network
  • to plug into physical network and access via lan, need to find the pi
  • to find the pi, need to install nmap on widy openwrt via opkg
  • to install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)
  • to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)
  • once all that is done, then we can put build-essential on the pi
  • once we have build essential we can put the jtag software on the pi
  • tjtag pi: [2]
  • wiring diagram: [3]



new router

new router:

  • website with database of embedded dev boards:
  • Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
  • Banana Pi R2 Link: [4]

blog posts in progress

  • Project Euler 172
  • Project Euler 502
  • bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap


Genealogy photos:

  • Photos cropped/organized by family
    • 2011
    • 2017
    • Rename scheme
    • Notes - A2k11
    • Notes - R2k11
    • Notes - A2k17
    • Notes - K2k17
    • Notes - R2k17
  • Send email to fam with link on Dropbox


  • Pauline and Bruce chapters
  • Historical research planning


2018/January/Notes Repositories


project euler

  • Project Euler:
    • Project Euler: Problem 172
    • Clean up problems 52-67

Project Euler/172

rubiks cube

2018/January/Rubiks Cube

desk reorg

Reorganize and rearrange desk

  • Move book boxes to room, move table from room to main area
  • Consolidate plastic tubs
  • Computer lab - 2 laptops + desktop
  • Switch to dropbear as the main router

master git repos

Master repo layout:

  • Create master repo for each organization
  • Each repo in org forms submodule
  • Utility scripts to update to latest, and to commit latest


  • Tripos bot
    • Work out small page latex
    • Get questions into latex format
    • Script to compile latex to pdf, Imagemagick latex to jpg
    • Create Rainbow Mind Machine bot
    • Create Twitter account
    • Authenticate
    • Set up and run on Rojo
    • Debug - it's just not tweeting.
  • Apollo Bot
    • Fix lengths of dialogue per speaker
    • Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
    • Get back online
  • Milton Bot
    • Seems okay... but still having some permission problem
    • Problem was, could not access keys, b/c not run from correct directory.
    • Need to fix
    • Get back online
  • Ginsberg Bot
    • Seems okay, but permission problem (see above)
    • Get back online

Tripos Bot



  • jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
  • lite/vanilla/docker

dotfile submodules/organization:

  • one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
  • organization over convenience - not as much sharing because generally static/fixed

debian deployment:

  • rojo copy dotfile changes over to debian repo
  • rojo apply changes, ensure nothing broken
  • jupiter apply changes, ensure nothing broken

mac deployment:

  • copy dotfiles from debian
  • copy python from debian
  • apply changes, ensure nothing broken

install scripts:


kali deployment and rpi deployment to be fixed later



  • 2 TB hard drive (ordered)
  • Debian (100 gb windows)
  • Kali via persistent live usb configured for Jupiter


  • Charlesreid1 connection
  • Anonymous layer PIA on dropbear
  • Anonymous browsing notes, Privoxy, &c.
  • Tinc VPN


  • netdata
  • network traffic

Networking tools:


Netdata setup:

  • Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
  • Netdata running on each, and Netdata instances accessible from on the network
  • Install Prometheus on jupiter
  • Connect Netdata on basilisk and morpheus to Prometheus on jupiter

VPN mesh network:

  • Tinc
  • Working okay on the home network.

Netdata security:

  • Set up vpn via tinc on rojo
  • Set up netdata on rojo
  • Bind netdata to vpn ip address only
  • Verify rojo netdata instance protected

Digital ocean guide - what role does nginx play?

Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.


Prometheus/Security data

  • Share netdata securely
  • VPN connection to server
  • Logs, network traffic, loading, pages, processes



blog posts

  • Knuth permutation generation
  • 4x4 Rubik's Cube: Part 1: Representations
  • 4x4 Rubik's Cube: Part 2: Permutations
  • 4x4 Rubik's Cube: Part 3: Cycles
  • 4x4 Rubik's Cube: Part 4: Orders

jupiter oopsie

recovering from an accidental chmod of the entire /usr/lib directory.

Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.

Then reinstall. Then no wireless. Then reinstall with network connection.

Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.

Now... just... giving up. Back to Ubuntu. Everything's fine now.

Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.

Airplane mode was stuck in "on" mode, no physical swtich.

Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade

kali live usb

Kali/Persistent USB

  • question: how to persist e.g. keyboard preferences or installed software
  • answer: no matter... figure out when we need to.

fixing widy


  • widy is working okay as the central router
  • need a backup if keeping it in place permanently
  • complete the widy pivot to usb
  • collectd
  • nmap, ettercap, aircrack, etc.