Deployment/New Node Checklist

From charlesreid1

  • Domain records set up
    • a name base directory to IP of server
    • cname www redirect
    • not scalable, but that's what aws is for.
  • Aptitude
    • apt get update
    • aptitude build scripts
  • Sysadmin stuff
    • Make non-root default user
  • SSH
    • No root login
  • SSL
    • Getting certificate, certifying self as owner of web domain
  • Docker


Domain Records

If you want example.com to point to your new server, log into your host provider and set up the following DNS records.

Set up an A Name record with host @ (this is the A Name equivalent of a wild card) and set its target to the IP address of your server.

Next set up a C Name record with host www (this rule will catch all www.domain.com) and set its target to domain.com. (trailing period is important).

Now, any requests to www.domain.com will be sent to domain.com instead, and any requests to domain.com will go to the IP address of your machine.

Users

See Unix/Sysadmin

Add a non-root user

Better approach: use adduser (it comes first alphabetically, so it is better than useradd): 

adduser --quiet charles
passwd charles
usermod -aG sudo charles

# prepare to ssh as that user
mkdir /home/charles/.ssh
chown charles:charles /home/charles/.ssh
chmod 700 /home/charles/.ssh
touch /home/charles/.ssh/authorized_keys
chmod 600 /home/charles/.ssh/authorized_keys

(If you seem to recall seeing useradd more often than adduser, it's because useradd is the correct choice for Docker containers and Dockerfiles.)

Once user is in sudo group, no need to add them to sudoers file.

Don't be stupid. Log in as your regular user and double check that you can sudo.

If desired, edit the authorized keys file and add your public key so you can have passwordless access to the machine via ssh.

Dotfiles

Wait until you create a user to install any dotfiles, of course. Root remains plain and uncontaminated.

Ensure git is installed: 

which git

Optionally: 

apt-get install -y git

Now get the dotfiles repo: 

git clone https://git.charlesreid1.com/dotfiles/debian.git
cd debian/
./pre_bootstrap.sh
./bootstrap.sh

Log out and log back in. 

sudo ./set_machine_name.sh

Log out and log back in. 

sudo ./install_packages.sh

SSH

SSHD Config

WARNING

Before you do the next step, make sure you keep at least one shell open and logged in to the machine in case you end up locking yourself out.

Set up the sshd config file, located in /etc/ssh/sshd_config.

Specifically, change PermitRootLogin yes to PermitRootLogin no.

Here's a sed one liner to do it: 

$ sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

then restart the sshd service: 

$ sudo service sshd restart

Or just run this script as sudo: 

#!/bin/sh
sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
service sshd restart

SSL

This covers how you get an SSL certificate to run encrypted services on a server.

Lets Encrypt

Getting a certificate for the domain associated with the new node:

Installing Certbot

CertBot directs me to run the following as sudo: 

#!/bin/sh

# install add-apt-repository command
apt-get -y install software-properties-common
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get -y install certbot

Obtaining a cert from the "webroot" plugin (of certbot) requires access to the root web directory.

That means, we actually have to have a web server set up. Oh, ok, hang on.

Side note: installing Apache to test certificates

Really quick, let's install an apache web server so we can turn on SSL with it and test out our certificates. 

sudo apt-get install apache2

This installs apache2 to the default location. We can now put a quick "hello world" html file into the web root, which is by default at /var/www/html. (If we don't know where it is, we can check the default apache config file which is in /etc/apache2/, and that will specify the web root location.)

Create a hello world page: 

$ echo "<h2>OHAI WERLD</h2>" > /var/www/html/index.html

May need to be sudo, since by default my /var/www/ was owned by root.

Now start the apache service: 

$ sudo service apache2 restart

Now you can visit your new machine's IP address in the web browser and see "OHAI WERLD".

Note that if you set up any A Name records for a domain to forward to your server, you can also check those now.

This Apache server does not have SSL set up yet, we'll do that once we generate the SSL certificates.

Note: check who the Apache user is by opening /etc/apache2/envvars and check the variable export APACHE_RUN_USER=www-data.

This is the user who should own our web directory /var/www/htdocs: 

$ sudo chown -R www-data:www-data /var/www

Certbot to generate certificates

Now we use Certbot to generate SSL certificates: https://certbot.eff.org/#ubuntuxenial-nginx

Use the cert only framework to generate a web root certificate (the kind you generate if you control the web root directory).

No Arguments: 

$ certbot certonly

This asks for some information, like an email and whether you agree with the Lets Encrypt EULA: https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf

This makes several references to ISRG, which is the Internet Security Research Group, a non-profit based in California that served as the certificate authority willing to sign Lets Encrypt certificates without charging absurd amounts of money: https://letsencrypt.org/isrg/

This asks for your domain names, be sure and enter both example.com and www.example.com.

Well... then the stinker couldn't interpret my web directory input.

With Arguments:

I had more luck with clearly stated error messages when I ran certbot with the various flags to automatically answer the questions above. 

$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party -d www.reidmachine.party
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for reidmachine.party
http-01 challenge for wwwreidmachine.party
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. wwwreidmachine.party (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: DNS problem: NXDOMAIN looking up A for wwwreidmachine.party, reidmachine.party (http-01): urn:acme:error:unknownHost :: The server could not resolve a domain name :: No valid IP addresses found for reidmachine.party

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: reidmachine.party
   Type:   unknownHost
   Detail: No valid IP addresses found for reidmachine.party

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: www.reidmachine.party
   Type:   connection
   Detail: DNS problem: NXDOMAIN looking up A for www.reidmachine.party

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

This was a problem with the DNS records (see top of page). The www. domain was not working.

I ran the certificate-making process on just the domain that was working (no prefix) and that worked fine: 

$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for reidmachine.party
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/reidmachine.party/fullchain.pem. Your cert
   will expire on 2017-06-23. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Once the DNS records were sorted out, and the www. prefix was working, I re-ran the command with both domains. This seamlessly added the new domain to the end of the existing certificate. Pretty slick! 

$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party -d www.reidmachine.party
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

-------------------------------------------------------------------------------
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/reidmachine.party.conf)

It contains these names: reidmachine.party

You requested these names for the new certificate: reidmachine.party,
www.reidmachine.party.

Do you want to expand and replace this existing certificate with the new
certificate?
-------------------------------------------------------------------------------
(E)xpand/(C)ancel: E
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for reidmachine.party
http-01 challenge for www.reidmachine.party
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/reidmachine.party/fullchain.pem. Your cert
   will expire on 2017-06-23. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Okay, so private and public key certificate file stuffs are here: 

/etc/letsencrypt/keys/0001_key-certbot.pem

Certbot cron job

Certbot package will run a cron job to renew certificates before expiry. The certificates last for 90 days. To do a dry run of a certificate renewal: 

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/reidmachine.party.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for reidmachine.party
http-01 challenge for www.reidmachine.party
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/reidmachine.party/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)

IMPORTANT NOTES:
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.


Docker

At this point, we're ready to install docker, and then load up the new node with whatever containers we need.

Installing

See Docker/Installing

Here is an official one-and-done bash script from Docker: https://git.charlesreid1.com/dotfiles/debian/src/master/get-docker.sh

Non-Sudo Docker Usage

Modify the non-root user to be able to run docker by adding them to the docker group: 

groupadd docker
usermod -a -G docker username

Docker Boats

Now you're all set up, and it's on to setting up your docker boats.

Once we've gone from "bare metal" to Docker, we're all set up to build a toolchain for whatever this node happens to be doing.

Next step is to build Docker boats - groups of containers - for different projects. See Docker/Boats.

Flag

Retrieved from "https://charlesreid1.com/w/index.php?title=Deployment/New_Node_Checklist&oldid=26818"