2018/January

Task list for January:

in progress

data engineering

Data engineering scenarios

• workflow with large amounts of data
• collectd on widy and others
• logging and log files
• dashboards
• Google Cloud/Review

network integration:

• everything is related...
• elasticsearch can search json, provides searchable interface to data
• log data into json format (mongo)
• collectd system metrics into json format (mongo)
• bot monitoring metrics, other metrics.... all into json format (mongo)
• then we just need to figure out the mongodb - grafana hookups

Master plan:

• the master plan is emerging
• elasticsearch link: [1]

2018/January/Data Engineering

kali networking

(on hold - tap hardware)

Network monitoring configuration setup:

Materials:

• isp-provided router (internet-enabled port)
• network tap device
• network monitoring server on a separate management lan
• "safe" gateway router for everybody

Set up the tap:

• port 1 of the tap connects to isp
• port 2 of the tap connects to safe router gateway, local wifi router
• port 3 of the tap connects to the network monitor, which is receiving network traffic capture from the tap device

Bro:

• Logs everything - but how to configure it? how to run it? what is the big picture?
• How does it fit into the network architecture - is it analyzing the pile of packets that's coming in to the security onion server?

Network monitoring distros:

• Security Onion
• Selks - consists of:
  • Suricata
  • Elasticsearch
  • Logstash
  • Kibana
  • Scirius
  • Evebox
• Rock Network Security Monitor
• What I'm trying to do: http://www.austintaylor.io/suricata/elasticsearch/logstash/continuous/monitoring/intrusion/detection/system/2016/12/17/build-a-world-class-monitoring-system-enterprise-small-office-home/

What this enables:

• hardware level interception point
• arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point

Next steps:

• Networks: MITM, wifi, evil ap, honeypot
• metasploit, sslstrip, certificate, frankencert
• focus on communications and networking
• pw credentials wall of sheep in a box

charlesreid1 wiki

scraping page links and history into database

• get a single page using Site() and Page() objects
• load the history of a single page using fullVersionHistory()
• get lists of all pages
• set up mongodb on jupiter
• scrape all templates
• pywikibot code link: https://pywikibot.readthedocs.io/en/latest/_modules/pywikibot/page/

Pywikibot

fixing dropbear

dropbear un-bricking:

• open the case
• to unbrick the router, we need to install stuff on the raspberry pi.
• to install stuff on the raspberry pi, we need to reflash the sd card.
• to reflash the sd card, we need to figure out which image to flash with. (used kali.)
• to get pi to join wifi, need to plug into physical network
• to plug into physical network and access via lan, need to find the pi
• to find the pi, need to install nmap on widy openwrt via opkg
• to install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)
• to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)
• once all that is done, then we can put build-essential on the pi
• once we have build essential we can put the jtag software on the pi
• tjtag pi: [2]
• wiring diagram: [3]

Dropbear

Dropbear/Unbricking

new router

new router:

• website with database of embedded dev boards: board-db.org
• Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
• Banana Pi R2 Link: [4]

blog posts in progress

• Project Euler 172
• Project Euler 502
• bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap

genealogy

Genealogy photos:

• Photos cropped/organized by family
  • 2011
  • 2017
  • Rename scheme
  • Notes - A2k11
  • Notes - R2k11
  • Notes - A2k17
  • Notes - K2k17
  • Notes - R2k17
• Send email to fam with link on Dropbox

Writing:

• Pauline and Bruce chapters
• Historical research planning

notes

• Shore up notes:
  • Experiment design: https://git.charlesreid1.com/notes/experiment-design
  • Linear models: https://git.charlesreid1.com/notes/linear-models
  • Rubiks cube: https://git.charlesreid1.com/notes/rubiks-notes

2018/January/Notes Repositories

complete

project euler

• Project Euler:
  • Project Euler: Problem 172
  • Clean up problems 52-67

Project Euler/172

rubiks cube

• Rubiks Cube:
  • Calculating the order of a permutation (see https://math.stackexchange.com/questions/332146/efficient-method-to-determine-the-order-of-a-permutation-in-s-n)

2018/January/Rubiks Cube

• Rubiks Cube/Numbers
• Rubiks Cube/Tuple
• Rubiks Cube/Permutations

desk reorg

Reorganize and rearrange desk

• Move book boxes to room, move table from room to main area
• Consolidate plastic tubs
• Computer lab - 2 laptops + desktop
• Switch to dropbear as the main router

master git repos

Master repo layout:

• Create master repo for each organization
• Each repo in org forms submodule
• Utility scripts to update to latest, and to commit latest

bots

• Tripos bot
  • Work out small page latex
  • Get questions into latex format
  • Script to compile latex to pdf, Imagemagick latex to jpg
  • Create Rainbow Mind Machine bot
  • Create Twitter account
  • Authenticate
  • Set up and run on Rojo
  • Debug - it's just not tweeting.

• Apollo Bot
  • Fix lengths of dialogue per speaker
  • Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
  • Get back online

• Milton Bot
  • Seems okay... but still having some permission problem
  • Problem was, could not access keys, b/c not run from correct directory.
  • Need to fix run_bots_in_screen.sh
  • Get back online

• Ginsberg Bot
  • Seems okay, but permission problem (see above)
  • Get back online

Tripos Bot

dotfiles

dotfiles:

• jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
• lite/vanilla/docker

dotfile submodules/organization:

• one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
• organization over convenience - not as much sharing because generally static/fixed

debian deployment:

• rojo copy dotfile changes over to debian repo
• rojo apply changes, ensure nothing broken
• jupiter apply changes, ensure nothing broken

mac deployment:

• copy dotfiles from debian
• copy python from debian
• apply changes, ensure nothing broken

install scripts:

• debian apt-get install script Jupiter
• mac brew install Homebrew/Post Install
• kali apt-get install Kali/Post Install  · Kali Raspberry Pi/Post Install
• rpi apt-get install RaspberryPi/Post Install

Dotfiles

• https://git.charlesreid1.com/dotfiles/vanilla
• https://git.charlesreid1.com/dotfiles/dotfiles-master
• https://git.charlesreid1.com/dotfiles/rpi
• https://git.charlesreid1.com/dotfiles/mac
• https://git.charlesreid1.com/dotfiles/kali
• https://git.charlesreid1.com/dotfiles/debian

kali deployment and rpi deployment to be fixed later

kali

Jupiter:

• 2 TB hard drive (ordered)
• Debian (100 gb windows)
• Kali via persistent live usb configured for Jupiter

VPN:

• Charlesreid1 connection
• Anonymous layer PIA on dropbear
• Anonymous browsing notes, Privoxy, &c.
• Tinc VPN

Instrumentation:

• netdata
• network traffic

Networking tools:

• Kali Tools installation and workflow
• Bro installation

netdata

Netdata setup:

• Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
• Netdata running on each, and Netdata instances accessible from on the network
• Install Prometheus on jupiter
• Connect Netdata on basilisk and morpheus to Prometheus on jupiter

VPN mesh network:

• Tinc
• Working okay on the home network.

Netdata security:

• Set up vpn via tinc on rojo
• Set up netdata on rojo
• Bind netdata to vpn ip address only
• Verify rojo netdata instance protected

Digital ocean guide - what role does nginx play?

Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.

Netdata/Security

Prometheus/Security

Charlesreid1.com data

• Share netdata securely
• VPN connection to server
• Logs, network traffic, loading, pages, processes

Netdata

Prometheus

blog posts

• Knuth permutation generation
• 4x4 Rubik's Cube: Part 1: Representations
• 4x4 Rubik's Cube: Part 2: Permutations
• 4x4 Rubik's Cube: Part 3: Cycles
• 4x4 Rubik's Cube: Part 4: Orders

jupiter oopsie

recovering from an accidental chmod of the entire /usr/lib directory.

Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.

Then reinstall. Then no wireless. Then reinstall with network connection.

Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.

Now... just... giving up. Back to Ubuntu. Everything's fine now.

Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.

Airplane mode was stuck in "on" mode, no physical swtich.

Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade

kali live usb

Kali/Persistent USB

• question: how to persist e.g. keyboard preferences or installed software
• answer: no matter... figure out when we need to.

fixing widy

widy:

• widy is working okay as the central router
• need a backup if keeping it in place permanently
• complete the widy pivot to usb
• collectd
• nmap, ettercap, aircrack, etc.

Widy