2018/January: Difference between revisions

From charlesreid1
Line 11: Line 11:


[[2018/January/Notes Repositories]]
[[2018/January/Notes Repositories]]
===netdata===
Netdata setup:
* Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
* Netdata running on each, and Netdata instances accessible from on the network
* Install Prometheus on jupiter
*
Understanding security issues/risks important
* Prometheus for time series
* Forget goddamn chemical reactors
* Focus on monitoring computational infrastructure, networking, security
Charlesreid1.com data
* Logs, network traffic, loading, pages, processes
* This is why we need to figure out SECURE Netdata and SECURE Prometheus
[[Netdata]]
[[Prometheus]]


===data engineering===
===data engineering===
Line 18: Line 40:
* See also: [[Google Cloud/Review]]
* See also: [[Google Cloud/Review]]


Logging
<s>Logging
* [[Netdata]]
* Get [[Netdata]] set up</s>
 
Network data
* Overarching goal
* Gather data
* Analyze trends
* Create data sets
* Data engineering: gathering data process, analysis process, modeling process
 
Charlesreid1.com data
* Logs
* Network traffic
* Loading
* Pages
* Processes


[[2018/January/Data Engineering]]
[[2018/January/Data Engineering]]

Revision as of 11:43, 19 January 2018

Task list for January:

planning stages

notes

2018/January/Notes Repositories

netdata

Netdata setup:

  • Install Netdata on jupiter, basilisk, morpheus (eventually, raspberry pis)
  • Netdata running on each, and Netdata instances accessible from on the network
  • Install Prometheus on jupiter


Understanding security issues/risks important

  • Prometheus for time series
  • Forget goddamn chemical reactors
  • Focus on monitoring computational infrastructure, networking, security

Charlesreid1.com data

  • Logs, network traffic, loading, pages, processes
  • This is why we need to figure out SECURE Netdata and SECURE Prometheus

Netdata

Prometheus

data engineering

Data engineering scenarios

  • Basic workflow, dealing with large amounts of data and doing machine learning on it
  • See also: Google Cloud/Review

Logging

2018/January/Data Engineering

kali

Jupiter:

  • 2 TB hard drive (ordered)
  • Debian (100 gb windows)
  • Kali via persistent live usb configured for Jupiter

Live USB problems:

OpenVPN:

  • Charlesreid1 connection
  • Anonymous layer PIA on dropbear
  • Anonymous browsing notes, Privoxy, &c.

Instrumentation:

  • netdata
  • network traffic

MITM:

  • MITMf
  • SSLStrip
  • urlsnarf, mailsnarf, etc - WOSIAB

General thread of things:

  • MITM
  • Windows exploits
  • Payload generation
  • Remote shells
  • C&C
  • Metasploitable, etc.

Git repos:

  • Setting up network testbed and instrumentation (W?)
  • Scripts for executing different attacks, etc.
  • Dropbear scripts - new org? kali?

raspberry pi

Widget. LCD/lights/something.

Wireless weatherproof IP camera?

in progress

mon jan 15

bots

master git repos

blog posts

  • Blog posts:
    • Knuth permutation generation
    • 4x4 Rubik's Cube: Part 1: Representations
    • 4x4 Rubik's Cube: Part 2: Permutations
    • 4x4 Rubik's Cube: Part 3: Cycles
    • Bots (RMM, Queneau)
    • Google Data Engineering Certification blog post and notes highlights
    • Data engineering scenario rollouts

genealogy

Genealogy photos:

  • Photos cropped/organized by family
    • 2011
    • 2017
    • Rename scheme
    • Notes document initialized
  • Send email to fam with link on Dropbox

Writing:

  • Pauline and Bruce chapters
  • Historical research planning

complete

project euler

  • Project Euler:
    • Project Euler: Problem 172
    • Clean up problems 52-67

Project Euler/172

rubiks cube

2018/January/Rubiks Cube

desk reorg

Reorganize and rearrange desk

  • Move book boxes to room, move table from room to main area
  • Consolidate plastic tubs
  • Computer lab - 2 laptops + desktop
  • Switch to dropbear as the main router

master git repos

Create master repo for each organization - each repo in org as a submodule

bots

  • Tripos bot
    • Work out small page latex
    • Get questions into latex format
    • Script to compile latex to pdf, Imagemagick latex to jpg
    • Create Rainbow Mind Machine bot
    • Create Twitter account
    • Authenticate
    • Set up and run on Rojo
    • Debug - it's just not tweeting.
  • Apollo Bot
    • Fix lengths of dialogue per speaker
    • Make lengths of dialogue per speaker a parameter/variable for function/library in rainbow mind machine
    • Get back online
  • Milton Bot
    • Seems okay... but still having some permission problem
    • Problem was, could not access keys, b/c not run from correct directory.
    • Need to fix run_bots_in_screen.sh
    • Get back online
  • Ginsberg Bot
    • Seems okay, but permission problem (see above)
    • Get back online

Tripos Bot

dotfiles

dotfiles:

  • jupiter/rojo, maya/cronus, basilisk/morpheus, thing1/thing2
  • lite/vanilla/docker

dotfile submodules/organization:

  • one repo per machine (or machine "class") - debian, mac, kali, raspberry pi
  • organization over convenience - not as much sharing because generally static/fixed

debian deployment:

  • rojo copy dotfile changes over to debian repo
  • rojo apply changes, ensure nothing broken
  • jupiter apply changes, ensure nothing broken

mac deployment:

  • copy dotfiles from debian
  • copy python from debian
  • apply changes, ensure nothing broken

kali deployment:

rpi deployment:

install scripts:

Dotfiles

Flags

Retrieved from "https://charlesreid1.com/w/index.php?title=2018/January&oldid=23390"