2018/January
From charlesreid1
Task list for January:
Contents
in progress
data engineering
Data engineering scenarios
- workflow with large amounts of data
- collectd on widy and others
- logging and log files
- dashboards
- Google Cloud/Review
network integration:
- everything is related...
- elasticsearch can search json, provides searchable interface to data
- log data into json format (mongo)
- collectd system metrics into json format (mongo)
- bot monitoring metrics, other metrics.... all into json format (mongo)
- then we just need to figure out the mongodb - grafana hookups
Master plan:
- the master plan is emerging
- elasticsearch link: [1]
kali networking
(on hold - tap hardware)
Network monitoring configuration setup:
Materials:
- isp-provided router (internet-enabled port)
- network tap device
- network monitoring server on a separate management lan
- "safe" gateway router for everybody
Set up the tap:
- port 1 of the tap connects to isp
- port 2 of the tap connects to safe router gateway, local wifi router
- port 3 of the tap connects to the network monitor, which is receiving network traffic capture from the tap device
Bro:
- Logs everything - but how to configure it? how to run it? what is the big picture?
- How does it fit into the network architecture - is it analyzing the pile of packets that's coming in to the security onion server?
Network monitoring distros:
- Security Onion
- Selks - consists of:
- Suricata
- Elasticsearch
- Logstash
- Kibana
- Scirius
- Evebox
- Rock Network Security Monitor
- What I'm trying to do: http://www.austintaylor.io/suricata/elasticsearch/logstash/continuous/monitoring/intrusion/detection/system/2016/12/17/build-a-world-class-monitoring-system-enterprise-small-office-home/
What this enables:
- hardware level interception point
- arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point
Next steps:
- Networks: MITM, wifi, evil ap, honeypot
- metasploit, sslstrip, certificate, frankencert
- focus on communications and networking
- pw credentials wall of sheep in a box
charlesreid1 wiki
scraping page links and history into database
get a single page using Site() and Page() objectsload the history of a single page using fullVersionHistory()get lists of all pages- set up mongodb on jupiter
- scrape all templates
- pywikibot code link: https://pywikibot.readthedocs.io/en/latest/_modules/pywikibot/page/
fixing dropbear
dropbear un-bricking:
open the caseto unbrick the router, we need to install stuff on the raspberry pi.to install stuff on the raspberry pi, we need to reflash the sd card.to reflash the sd card, we need to figure out which image to flash with. (used kali.)to get pi to join wifi, need to plug into physical networkto plug into physical network and access via lan, need to find the pito find the pi, need to install nmap on widy openwrt via opkgto install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)once all that is done, then we can put build-essential on the pionce we have build essential we can put the jtag software on the pi- tjtag pi: [2]
- wiring diagram: [3]
new router
new router:
- website with database of embedded dev boards: board-db.org
- Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
- Banana Pi R2 Link: [4]
blog posts in progress
- Project Euler 172
- Project Euler 502
- bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap
genealogy
Genealogy photos:
- Photos cropped/organized by family
20112017Rename schemeNotes - A2k11- Notes - R2k11
- Notes - A2k17
- Notes - K2k17
- Notes - R2k17
- Send email to fam with link on Dropbox
Writing:
- Pauline and Bruce chapters
- Historical research planning
notes
- Shore up notes:
- Experiment design: https://git.charlesreid1.com/notes/experiment-design
- Linear models: https://git.charlesreid1.com/notes/linear-models
- Rubiks cube: https://git.charlesreid1.com/notes/rubiks-notes
2018/January/Notes Repositories
complete
project euler
Project Euler:Project Euler: Problem 172Clean up problems 52-67
rubiks cube
Rubiks Cube:Calculating the order of a permutation (see https://math.stackexchange.com/questions/332146/efficient-method-to-determine-the-order-of-a-permutation-in-s-n)
desk reorg
Reorganize and rearrange desk
Move book boxes to room, move table from room to main areaConsolidate plastic tubsComputer lab - 2 laptops + desktopSwitch to dropbear as the main router
master git repos
Master repo layout:
- Create master repo for each organization
- Each repo in org forms submodule
Utility scripts to update to latest, and to commit latest
bots
Tripos botWork out small page latexGet questions into latex formatScript to compile latex to pdf, Imagemagick latex to jpgCreate Rainbow Mind Machine botCreate Twitter accountAuthenticateSet up and run on RojoDebug - it's just not tweeting.
Apollo BotFix lengths of dialogue per speakerMake lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machineGet back online
Milton BotSeems okay... but still having some permission problem- Problem was, could not access keys, b/c not run from correct directory.
Need to fix run_bots_in_screen.shGet back online
Ginsberg BotSeems okay, but permission problem(see above)Get back online
dotfiles
dotfiles:
- jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
- lite/vanilla/docker
dotfile submodules/organization:
- one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
organization over convenience - not as much sharing because generally static/fixed
debian deployment:
- rojo copy dotfile changes over to debian repo
- rojo apply changes, ensure nothing broken
jupiter apply changes, ensure nothing broken
mac deployment:
- copy dotfiles from debian
- copy python from debian
apply changes, ensure nothing broken
install scripts:
- debian apt-get install script Jupiter
- mac brew install Homebrew/Post Install
- kali apt-get install Kali/Post Install · Kali Raspberry Pi/Post Install
rpi apt-get install RaspberryPi/Post Install
- https://git.charlesreid1.com/dotfiles/vanilla
- https://git.charlesreid1.com/dotfiles/dotfiles-master
- https://git.charlesreid1.com/dotfiles/rpi
- https://git.charlesreid1.com/dotfiles/mac
- https://git.charlesreid1.com/dotfiles/kali
- https://git.charlesreid1.com/dotfiles/debian
kali deployment and rpi deployment to be fixed later
kali
Jupiter:
2 TB hard drive(ordered)Debian (100 gb windows)Kali via persistent live usb configured for Jupiter
VPN:
- Charlesreid1 connection
- Anonymous layer PIA on dropbear
- Anonymous browsing notes, Privoxy, &c.
Tinc VPN
Instrumentation:
- netdata
network traffic
Networking tools:
- Kali Tools installation and workflow
Bro installation
netdata
Netdata setup:
- Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
- Netdata running on each, and Netdata instances accessible from on the network
- Install Prometheus on jupiter
Connect Netdata on basilisk and morpheus to Prometheus on jupiter
VPN mesh network:
- Tinc
Working okay on the home network.
Netdata security:
- Set up vpn via tinc on rojo
- Set up netdata on rojo
- Bind netdata to vpn ip address only
Verify rojo netdata instance protected
Digital ocean guide - what role does nginx play?
Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.
Charlesreid1.com data
- Share netdata securely
- VPN connection to server
Logs, network traffic, loading, pages, processes
blog posts
Knuth permutation generation4x4 Rubik's Cube: Part 1: Representations4x4 Rubik's Cube: Part 2: Permutations4x4 Rubik's Cube: Part 3: Cycles4x4 Rubik's Cube: Part 4: Orders
jupiter oopsie
recovering from an accidental chmod of the entire /usr/lib directory.
Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.
Then reinstall. Then no wireless. Then reinstall with network connection.
Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.
Now... just... giving up. Back to Ubuntu. Everything's fine now.
Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.
Airplane mode was stuck in "on" mode, no physical swtich.
Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade
kali live usb
- question: how to persist e.g. keyboard preferences or installed software
answer: no matter... figure out when we need to.
fixing widy
widy:
widy is working okay as the central router- need a backup if keeping it in place permanently
- complete the widy pivot to usb
- collectd
nmap, ettercap, aircrack, etc.