Task list for January:
- 1 in progress
- 2 complete
- 3 Flags
Data engineering scenarios
- workflow with large amounts of data
- collectd on widy and others
- logging and log files
- Google Cloud/Review
- everything is related...
- elasticsearch can search json, provides searchable interface to data
- log data into json format (mongo)
- collectd system metrics into json format (mongo)
- bot monitoring metrics, other metrics.... all into json format (mongo)
- then we just need to figure out the mongodb - grafana hookups
- the master plan is emerging
- elasticsearch link: 
(on hold - tap hardware)
Network monitoring configuration setup:
- isp-provided router (internet-enabled port)
- network tap device
- network monitoring server on a separate management lan
- "safe" gateway router for everybody
Set up the tap:
- port 1 of the tap connects to isp
- port 2 of the tap connects to safe router gateway, local wifi router
- port 3 of the tap connects to the network monitor, which is receiving network traffic capture from the tap device
- Logs everything - but how to configure it? how to run it? what is the big picture?
- How does it fit into the network architecture - is it analyzing the pile of packets that's coming in to the security onion server?
Network monitoring distros:
- Security Onion
- Selks - consists of:
- Rock Network Security Monitor
- What I'm trying to do: http://www.austintaylor.io/suricata/elasticsearch/logstash/continuous/monitoring/intrusion/detection/system/2016/12/17/build-a-world-class-monitoring-system-enterprise-small-office-home/
What this enables:
- hardware level interception point
- arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point
- Networks: MITM, wifi, evil ap, honeypot
- metasploit, sslstrip, certificate, frankencert
- focus on communications and networking
- pw credentials wall of sheep in a box
scraping page links and history into database
get a single page using Site() and Page() objects
load the history of a single page using fullVersionHistory()
get lists of all pages
- set up mongodb on jupiter
- scrape all templates
- pywikibot code link: https://pywikibot.readthedocs.io/en/latest/_modules/pywikibot/page/
open the case
to unbrick the router, we need to install stuff on the raspberry pi.
to install stuff on the raspberry pi, we need to reflash the sd card.
to reflash the sd card, we need to figure out which image to flash with. (used kali.)
to get pi to join wifi, need to plug into physical network
to plug into physical network and access via lan, need to find the pi
to find the pi, need to install nmap on widy openwrt via opkg
to install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)
to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)
once all that is done, then we can put build-essential on the pi
once we have build essential we can put the jtag software on the pi
- tjtag pi: 
- wiring diagram: 
- website with database of embedded dev boards: board-db.org
- Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
- Banana Pi R2 Link: 
blog posts in progress
- Project Euler 172
- Project Euler 502
- bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap
- Photos cropped/organized by family
Notes - A2k11
- Notes - R2k11
- Notes - A2k17
- Notes - K2k17
- Notes - R2k17
- Send email to fam with link on Dropbox
- Pauline and Bruce chapters
- Historical research planning
- Shore up notes:
- Experiment design: https://charlesreid1.com:3000/notes/experiment-design
- Linear models: https://charlesreid1.com:3000/notes/linear-models
- Rubiks cube: https://charlesreid1.com:3000/notes/rubiks-notes
Project Euler: Problem 172
Clean up problems 52-67
Calculating the order of a permutation (see https://math.stackexchange.com/questions/332146/efficient-method-to-determine-the-order-of-a-permutation-in-s-n)
Reorganize and rearrange desk
Move book boxes to room, move table from room to main area
Consolidate plastic tubs
Computer lab - 2 laptops + desktop
Switch to dropbear as the main router
master git repos
Master repo layout:
- Create master repo for each organization
- Each repo in org forms submodule
- Utility scripts to update to latest, and to commit latest
Work out small page latex
Get questions into latex format
Script to compile latex to pdf, Imagemagick latex to jpg
Create Rainbow Mind Machine bot
Create Twitter account
Set up and run on Rojo
Debug - it's just not tweeting.
Fix lengths of dialogue per speaker
- Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
Get back online
Seems okay... but still having some permission problem
- Problem was, could not access keys, b/c not run from correct directory.
- Need to fix run_bots_in_screen.sh
Get back online
Seems okay, but permission problem(see above)
Get back online
- jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
- one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
- organization over convenience - not as much sharing because generally static/fixed
- rojo copy dotfile changes over to debian repo
- rojo apply changes, ensure nothing broken
- jupiter apply changes, ensure nothing broken
- copy dotfiles from debian
- copy python from debian
- apply changes, ensure nothing broken
- debian apt-get install script Jupiter
- mac brew install Homebrew/Post Install
- kali apt-get install Kali/Post Install · Kali Raspberry Pi/Post Install
- rpi apt-get install RaspberryPi/Post Install
kali deployment and rpi deployment to be fixed later
2 TB hard drive(ordered)
Debian (100 gb windows)
Kali via persistent live usb configured for Jupiter VPN:
- Charlesreid1 connection
- Anonymous layer PIA on dropbear
- Anonymous browsing notes, Privoxy, &c.
- Tinc VPN
- network traffic
- Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
- Netdata running on each, and Netdata instances accessible from on the network
- Install Prometheus on jupiter
- Connect Netdata on basilisk and morpheus to Prometheus on jupiter
VPN mesh network:
- Working okay on the home network.
- Set up vpn via tinc on rojo
- Set up netdata on rojo
- Bind netdata to vpn ip address only
- Verify rojo netdata instance protected
Digital ocean guide - what role does nginx play?
Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.
- Share netdata securely
- VPN connection to server
- Logs, network traffic, loading, pages, processes
Knuth permutation generation
4x4 Rubik's Cube: Part 1: Representations
4x4 Rubik's Cube: Part 2: Permutations
4x4 Rubik's Cube: Part 3: Cycles
4x4 Rubik's Cube: Part 4: Orders
recovering from an accidental chmod of the entire /usr/lib directory.
Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.
Then reinstall. Then no wireless. Then reinstall with network connection.
Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.
Now... just... giving up. Back to Ubuntu. Everything's fine now.
Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.
Airplane mode was stuck in "on" mode, no physical swtich.
Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade
kali live usb
- question: how to persist e.g. keyboard preferences or installed software
- answer: no matter... figure out when we need to.
widy is working okay as the central router
- need a backup if keeping it in place permanently
- complete the widy pivot to usb
- nmap, ettercap, aircrack, etc.