2018/January: Difference between revisions

From charlesreid1
m (Replacing charlesreid1.com:3000 with git.charlesreid1.com)
 
(30 intermediate revisions by the same user not shown)
Line 7: Line 7:
Data engineering scenarios
Data engineering scenarios
* workflow with large amounts of data
* workflow with large amounts of data
* collectd on widy and others
* logging and log files
* logging and log files
* dashboards
* dashboards
* [[Google Cloud/Review]]
* [[Google Cloud/Review]]


[[2018/January/Data Engineering]]
network integration:
* everything is related...
* elasticsearch can search json, provides searchable interface to data
* log data into json format (mongo)
* collectd system metrics into json format (mongo)
* bot monitoring metrics, other metrics.... all into json format (mongo)
* then we just need to figure out the mongodb - grafana hookups


===charlesreid1 wiki===
Master plan:
* the master plan is emerging
* elasticsearch link: [https://www.sitepoint.com/building-recipe-search-site-angular-elasticsearch/]


scraping page links and history into database
[[2018/January/Data Engineering]]
* depends on mongodb on jupiter
* wiki: https://pywikibot.readthedocs.io/en/latest/_modules/pywikibot/page/ - fullVersionHistory()
* [[Pywikibot]]


===router===
===kali networking===
 
widy working ok. would want a backup if keeping widy in place permanently.
 
Website with database of embedded dev boards: [https://www.board-db.org/search.php?q=&ram_min=&cpu_speed_min=&cpu_cores=&cpu_arch=any&price_min=&price_max=&storage_min=&gpio_min=&usb_min=&type=&lan=on&multiple_lan=on&lan_speed=0&order=price&order_d=a#results board-db.org]
 
The Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
 
Link: https://www.aliexpress.com/item/BPI-R2-Banana-PI-R2-Smart-Open-source-Wireless-Router-BPI-R2-Smart-Home-Control-Device/32825598599.html
 
Short term, need to use jtag to reflash [[Dropbear]]
 
Collectd on Widy
 
Widy USB pivot - complete
 
===kali live usb===


[[Kali/Persistent USB]]
(on hold - tap hardware)
* question: how to persist e.g. keyboard preferences or installed software
 
===kali networking===


Network monitoring configuration setup:
Network monitoring configuration setup:
Line 48: Line 35:
* isp-provided router (internet-enabled port)
* isp-provided router (internet-enabled port)
* network tap device
* network tap device
* security onion/network monitoring server on a separate management lan
* network monitoring server on a separate management lan
* "safe" gateway router for everybody
* "safe" gateway router for everybody


Line 73: Line 60:


What this enables:
What this enables:
* Now, instrumentation is in place to see what arp spoofing, MITM, honeypot, &c. look like
* hardware level interception point
* arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point


Next steps:
Next steps:
Line 80: Line 68:
* focus on communications and networking
* focus on communications and networking
* pw credentials wall of sheep in a box
* pw credentials wall of sheep in a box
===charlesreid1 wiki===
scraping page links and history into database
* <s>get a single page using Site() and Page() objects</s>
* <s>load the history of a single page using fullVersionHistory()</s>
* <s>get lists of all pages</s>
* set up mongodb on jupiter
* scrape all templates
* pywikibot code link: https://pywikibot.readthedocs.io/en/latest/_modules/pywikibot/page/
[[Pywikibot]]
===fixing dropbear===
dropbear un-bricking:
* <s>open the case</s>
* <s>to unbrick the router, we need to install stuff on the raspberry pi.</s>
* <s>to install stuff on the raspberry pi, we need to reflash the sd card.</s>
* <s>to reflash the sd card, we need to figure out which image to flash with. (used kali.)</s>
* <s>to get pi to join wifi, need to plug into physical network</s>
* <s>to plug into physical network and access via lan, need to find the pi</s>
* <s>to find the pi, need to install nmap on widy openwrt via opkg</s>
* <s>to install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)</s>
* <s>to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)</s>
* <s>once all that is done, then we can put build-essential on the pi</s>
* <s>once we have build essential we can put the jtag software on the pi</s>
* tjtag pi: [https://github.com/oxplot/tjtag-pi]
* wiring diagram: [https://github.com/oxplot/tjtag-pi/blob/master/wiring.jpg]
[[Dropbear]]
[[Dropbear/Unbricking]]
===new router===
new router:
* website with database of embedded dev boards: [https://www.board-db.org/search.php?q=&ram_min=&cpu_speed_min=&cpu_cores=&cpu_arch=any&price_min=&price_max=&storage_min=&gpio_min=&usb_min=&type=&lan=on&multiple_lan=on&lan_speed=0&order=price&order_d=a#results board-db.org]
* Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
* Banana Pi R2 Link: [https://www.aliexpress.com/item/BPI-R2-Banana-PI-R2-Smart-Open-source-Wireless-Router-BPI-R2-Smart-Home-Control-Device/32825598599.html]


===blog posts in progress===
===blog posts in progress===
Line 108: Line 136:


* Shore up notes:
* Shore up notes:
** Experiment design: https://charlesreid1.com:3000/notes/experiment-design
** Experiment design: https://git.charlesreid1.com/notes/experiment-design
** Linear models: https://charlesreid1.com:3000/notes/linear-models
** Linear models: https://git.charlesreid1.com/notes/linear-models
** Rubiks cube: https://charlesreid1.com:3000/notes/rubiks-notes
** Rubiks cube: https://git.charlesreid1.com/notes/rubiks-notes


[[2018/January/Notes Repositories]]
[[2018/January/Notes Repositories]]
Line 206: Line 234:


[[Dotfiles]]
[[Dotfiles]]
* https://charlesreid1.com:3000/dotfiles/vanilla
* https://git.charlesreid1.com/dotfiles/vanilla
* https://charlesreid1.com:3000/dotfiles/dotfiles-master
* https://git.charlesreid1.com/dotfiles/dotfiles-master
* https://charlesreid1.com:3000/dotfiles/rpi
* https://git.charlesreid1.com/dotfiles/rpi
* https://charlesreid1.com:3000/dotfiles/mac
* https://git.charlesreid1.com/dotfiles/mac
* https://charlesreid1.com:3000/dotfiles/kali
* https://git.charlesreid1.com/dotfiles/kali
* https://charlesreid1.com:3000/dotfiles/debian
* https://git.charlesreid1.com/dotfiles/debian


kali deployment and rpi deployment to be fixed later
kali deployment and rpi deployment to be fixed later
Line 278: Line 306:
* <s>4x4 Rubik's Cube: Part 3: Cycles</s>
* <s>4x4 Rubik's Cube: Part 3: Cycles</s>
* <s>4x4 Rubik's Cube: Part 4: Orders</s>
* <s>4x4 Rubik's Cube: Part 4: Orders</s>
===jupiter oopsie===
<s>recovering from an accidental chmod of the entire /usr/lib directory.
Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.
Then reinstall. Then no wireless. Then reinstall with network connection.
Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.
Now... just... giving up. Back to Ubuntu. Everything's fine now.
Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.
Airplane mode was stuck in "on" mode, no physical swtich.
Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade</s>
===kali live usb===
<s>[[Kali/Persistent USB]]
* question: how to persist e.g. keyboard preferences or installed software
* answer: no matter... figure out when we need to.</s>
===fixing widy===
widy:
* <s>widy is working okay as the central router
* need a backup if keeping it in place permanently
* complete the widy pivot to usb
* collectd
* nmap, ettercap, aircrack, etc.</s>
[[Widy]]




Latest revision as of 03:09, 9 October 2019

Task list for January:

in progress

data engineering

Data engineering scenarios

  • workflow with large amounts of data
  • collectd on widy and others
  • logging and log files
  • dashboards
  • Google Cloud/Review

network integration:

  • everything is related...
  • elasticsearch can search json, provides searchable interface to data
  • log data into json format (mongo)
  • collectd system metrics into json format (mongo)
  • bot monitoring metrics, other metrics.... all into json format (mongo)
  • then we just need to figure out the mongodb - grafana hookups

Master plan:

  • the master plan is emerging
  • elasticsearch link: [1]

2018/January/Data Engineering

kali networking

(on hold - tap hardware)

Network monitoring configuration setup:

Materials:

  • isp-provided router (internet-enabled port)
  • network tap device
  • network monitoring server on a separate management lan
  • "safe" gateway router for everybody

Set up the tap:

  • port 1 of the tap connects to isp
  • port 2 of the tap connects to safe router gateway, local wifi router
  • port 3 of the tap connects to the network monitor, which is receiving network traffic capture from the tap device

Bro:

  • Logs everything - but how to configure it? how to run it? what is the big picture?
  • How does it fit into the network architecture - is it analyzing the pile of packets that's coming in to the security onion server?

Network monitoring distros:

What this enables:

  • hardware level interception point
  • arp spoofing, mitm, honeypot, etc. - both an attack point and an observation point

Next steps:

  • Networks: MITM, wifi, evil ap, honeypot
  • metasploit, sslstrip, certificate, frankencert
  • focus on communications and networking
  • pw credentials wall of sheep in a box

charlesreid1 wiki

scraping page links and history into database

Pywikibot

fixing dropbear

dropbear un-bricking:

  • open the case
  • to unbrick the router, we need to install stuff on the raspberry pi.
  • to install stuff on the raspberry pi, we need to reflash the sd card.
  • to reflash the sd card, we need to figure out which image to flash with. (used kali.)
  • to get pi to join wifi, need to plug into physical network
  • to plug into physical network and access via lan, need to find the pi
  • to find the pi, need to install nmap on widy openwrt via opkg
  • to install nmap on widy, need to resolve bunch of missing opkg packags (resolved: packages and oldpackages split)
  • to get pi up and running and cleaned up, need to revisit old kali dotfiles (slew of work to do with new dotfiles repositories for kali raspberry pi)
  • once all that is done, then we can put build-essential on the pi
  • once we have build essential we can put the jtag software on the pi
  • tjtag pi: [2]
  • wiring diagram: [3]

Dropbear

Dropbear/Unbricking

new router

new router:

  • website with database of embedded dev boards: board-db.org
  • Banana Pi R2 is designed with built-in switch hardware, so it's intended to be used as a Raspberry Pi for home routers, of sorts. Long term, this would be a good hardware platform.
  • Banana Pi R2 Link: [4]

blog posts in progress

  • Project Euler 172
  • Project Euler 502
  • bootstrap table presenting results of Rubik's Cube investigation - Rubiks Cube/Bootstrap

genealogy

Genealogy photos:

  • Photos cropped/organized by family
    • 2011
    • 2017
    • Rename scheme
    • Notes - A2k11
    • Notes - R2k11
    • Notes - A2k17
    • Notes - K2k17
    • Notes - R2k17
  • Send email to fam with link on Dropbox

Writing:

  • Pauline and Bruce chapters
  • Historical research planning

notes

2018/January/Notes Repositories

complete

project euler

  • Project Euler:
    • Project Euler: Problem 172
    • Clean up problems 52-67

Project Euler/172

rubiks cube

2018/January/Rubiks Cube

desk reorg

Reorganize and rearrange desk

  • Move book boxes to room, move table from room to main area
  • Consolidate plastic tubs
  • Computer lab - 2 laptops + desktop
  • Switch to dropbear as the main router

master git repos

Master repo layout:

  • Create master repo for each organization
  • Each repo in org forms submodule
  • Utility scripts to update to latest, and to commit latest

bots

  • Tripos bot
    • Work out small page latex
    • Get questions into latex format
    • Script to compile latex to pdf, Imagemagick latex to jpg
    • Create Rainbow Mind Machine bot
    • Create Twitter account
    • Authenticate
    • Set up and run on Rojo
    • Debug - it's just not tweeting.
  • Apollo Bot
    • Fix lengths of dialogue per speaker
    • Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
    • Get back online
  • Milton Bot
    • Seems okay... but still having some permission problem
    • Problem was, could not access keys, b/c not run from correct directory.
    • Need to fix run_bots_in_screen.sh
    • Get back online
  • Ginsberg Bot
    • Seems okay, but permission problem (see above)
    • Get back online

Tripos Bot

dotfiles

dotfiles:

  • jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
  • lite/vanilla/docker

dotfile submodules/organization:

  • one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
  • organization over convenience - not as much sharing because generally static/fixed

debian deployment:

  • rojo copy dotfile changes over to debian repo
  • rojo apply changes, ensure nothing broken
  • jupiter apply changes, ensure nothing broken

mac deployment:

  • copy dotfiles from debian
  • copy python from debian
  • apply changes, ensure nothing broken

install scripts:

Dotfiles

kali deployment and rpi deployment to be fixed later

kali

Jupiter:

  • 2 TB hard drive (ordered)
  • Debian (100 gb windows)
  • Kali via persistent live usb configured for Jupiter

VPN:

  • Charlesreid1 connection
  • Anonymous layer PIA on dropbear
  • Anonymous browsing notes, Privoxy, &c.
  • Tinc VPN

Instrumentation:

  • netdata
  • network traffic

Networking tools:

netdata

Netdata setup:

  • Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
  • Netdata running on each, and Netdata instances accessible from on the network
  • Install Prometheus on jupiter
  • Connect Netdata on basilisk and morpheus to Prometheus on jupiter

VPN mesh network:

  • Tinc
  • Working okay on the home network.

Netdata security:

  • Set up vpn via tinc on rojo
  • Set up netdata on rojo
  • Bind netdata to vpn ip address only
  • Verify rojo netdata instance protected

Digital ocean guide - what role does nginx play?

Focus on monitoring computational infrastructure, networking, security. Focus on DASHBOARDS.

Netdata/Security

Prometheus/Security

Charlesreid1.com data

  • Share netdata securely
  • VPN connection to server
  • Logs, network traffic, loading, pages, processes

Netdata

Prometheus

blog posts

  • Knuth permutation generation
  • 4x4 Rubik's Cube: Part 1: Representations
  • 4x4 Rubik's Cube: Part 2: Permutations
  • 4x4 Rubik's Cube: Part 3: Cycles
  • 4x4 Rubik's Cube: Part 4: Orders

jupiter oopsie

recovering from an accidental chmod of the entire /usr/lib directory.

Oops. Good thing it wasn't on Rojo. Aaaaand, a good reason to have a partition for the home directory.

Then reinstall. Then no wireless. Then reinstall with network connection.

Then back to the same stupid problems I've been having. Non-root user, not a sudoer, correct drivers not included. All stupid Debian stuff.

Now... just... giving up. Back to Ubuntu. Everything's fine now.

Meanwhile, on morpheus, I had installed security onion, then given up and gone back to kali, and wireless also broke.

Airplane mode was stuck in "on" mode, no physical swtich.

Morpheus took a sudo apt-get update and sudo apt-get upgrade and a sudo apt-get dist-upgrade

kali live usb

Kali/Persistent USB

  • question: how to persist e.g. keyboard preferences or installed software
  • answer: no matter... figure out when we need to.

fixing widy

widy:

  • widy is working okay as the central router
  • need a backup if keeping it in place permanently
  • complete the widy pivot to usb
  • collectd
  • nmap, ettercap, aircrack, etc.

Widy


Flags

Retrieved from "https://charlesreid1.com/w/index.php?title=2018/January&oldid=26796"