Deployment/New Node Checklist
From charlesreid1
- Domain records set up
- a name base directory to IP of server
- cname www redirect
- not scalable, but that's what aws is for.
- Aptitude
- apt get update
- aptitude build scripts
- Sysadmin stuff
- Make non-root default user
- SSH
- No root login
- SSL
- Getting certificate, certifying self as owner of web domain
- Docker
Domain Records
If you want example.com to point to your new server, log into your host provider and set up the following DNS records.
Set up an A Name record with host @
(this is the A Name equivalent of a wild card) and set its target to the IP address of your server.
Next set up a C Name record with host www
(this rule will catch all www.domain.com) and set its target to domain.com.
(trailing period is important).
Now, any requests to www.domain.com will be sent to domain.com instead, and any requests to domain.com will go to the IP address of your machine.
Users
See Unix/Sysadmin
Add a non-root user
Better approach: use adduser (it comes first alphabetically, so it is better than useradd):
adduser --quiet charles passwd charles usermod -aG sudo charles # prepare to ssh as that user mkdir /home/charles/.ssh chown charles:charles /home/charles/.ssh chmod 700 /home/charles/.ssh touch /home/charles/.ssh/authorized_keys chmod 600 /home/charles/.ssh/authorized_keys
(If you seem to recall seeing useradd more often than adduser, it's because useradd is the correct choice for Docker containers and Dockerfiles.)
Once user is in sudo group, no need to add them to sudoers file.
Don't be stupid. Log in as your regular user and double check that you can sudo.
If desired, edit the authorized keys file and add your public key so you can have passwordless access to the machine via ssh.
Dotfiles
Wait until you create a user to install any dotfiles, of course. Root remains plain and uncontaminated.
Ensure git is installed:
which git
Optionally:
apt-get install -y git
Now get the dotfiles repo:
git clone https://git.charlesreid1.com/dotfiles/debian.git cd debian/ ./pre_bootstrap.sh ./bootstrap.sh
Log out and log back in.
sudo ./set_machine_name.sh
Log out and log back in.
sudo ./install_packages.sh
SSH
SSHD Config
WARNING
Before you do the next step, make sure you keep at least one shell open and logged in to the machine in case you end up locking yourself out.
Set up the sshd config file, located in /etc/ssh/sshd_config
.
Specifically, change PermitRootLogin yes
to PermitRootLogin no
.
Here's a sed one liner to do it:
$ sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
then restart the sshd service:
$ sudo service sshd restart
Or just run this script as sudo:
#!/bin/sh sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config service sshd restart
SSL
This covers how you get an SSL certificate to run encrypted services on a server.
Lets Encrypt
Getting a certificate for the domain associated with the new node:
- Visit Let's Encrypt website https://letsencrypt.org/
- If command line access, redirected to CertBot https://certbot.eff.org/
- Run commands to install certbot command line utility
Installing Certbot
CertBot directs me to run the following as sudo:
#!/bin/sh # install add-apt-repository command apt-get -y install software-properties-common add-apt-repository ppa:certbot/certbot apt-get update apt-get -y install certbot
Obtaining a cert from the "webroot" plugin (of certbot) requires access to the root web directory.
That means, we actually have to have a web server set up. Oh, ok, hang on.
Side note: installing Apache to test certificates
Really quick, let's install an apache web server so we can turn on SSL with it and test out our certificates.
sudo apt-get install apache2
This installs apache2 to the default location. We can now put a quick "hello world" html file into the web root, which is by default at /var/www/html
. (If we don't know where it is, we can check the default apache config file which is in /etc/apache2/
, and that will specify the web root location.)
Create a hello world page:
$ echo "<h2>OHAI WERLD</h2>" > /var/www/html/index.html
May need to be sudo, since by default my /var/www/
was owned by root.
Now start the apache service:
$ sudo service apache2 restart
Now you can visit your new machine's IP address in the web browser and see "OHAI WERLD".
Note that if you set up any A Name records for a domain to forward to your server, you can also check those now.
This Apache server does not have SSL set up yet, we'll do that once we generate the SSL certificates.
Note: check who the Apache user is by opening /etc/apache2/envvars
and check the variable export APACHE_RUN_USER=www-data
.
This is the user who should own our web directory /var/www/htdocs
:
$ sudo chown -R www-data:www-data /var/www
Certbot to generate certificates
Now we use Certbot to generate SSL certificates: https://certbot.eff.org/#ubuntuxenial-nginx
Use the cert only framework to generate a web root certificate (the kind you generate if you control the web root directory).
No Arguments:
$ certbot certonly
This asks for some information, like an email and whether you agree with the Lets Encrypt EULA: https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf
This makes several references to ISRG, which is the Internet Security Research Group, a non-profit based in California that served as the certificate authority willing to sign Lets Encrypt certificates without charging absurd amounts of money: https://letsencrypt.org/isrg/
This asks for your domain names, be sure and enter both example.com
and www.example.com
.
Well... then the stinker couldn't interpret my web directory input.
With Arguments:
I had more luck with clearly stated error messages when I ran certbot with the various flags to automatically answer the questions above.
$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party -d www.reidmachine.party Saving debug log to /var/log/letsencrypt/letsencrypt.log Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: http-01 challenge for reidmachine.party http-01 challenge for wwwreidmachine.party Using the webroot path /var/www/html for all unmatched domains. Waiting for verification... Cleaning up challenges Failed authorization procedure. wwwreidmachine.party (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: DNS problem: NXDOMAIN looking up A for wwwreidmachine.party, reidmachine.party (http-01): urn:acme:error:unknownHost :: The server could not resolve a domain name :: No valid IP addresses found for reidmachine.party IMPORTANT NOTES: - The following errors were reported by the server: Domain: reidmachine.party Type: unknownHost Detail: No valid IP addresses found for reidmachine.party To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. - The following errors were reported by the server: Domain: www.reidmachine.party Type: connection Detail: DNS problem: NXDOMAIN looking up A for www.reidmachine.party To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
This was a problem with the DNS records (see top of page). The www. domain was not working.
I ran the certificate-making process on just the domain that was working (no prefix) and that worked fine:
$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party Saving debug log to /var/log/letsencrypt/letsencrypt.log Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: http-01 challenge for reidmachine.party Using the webroot path /var/www/html for all unmatched domains. Waiting for verification... Cleaning up challenges Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/reidmachine.party/fullchain.pem. Your cert will expire on 2017-06-23. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
Once the DNS records were sorted out, and the www. prefix was working, I re-ran the command with both domains. This seamlessly added the new domain to the end of the existing certificate. Pretty slick!
$ sudo certbot certonly --webroot -w /var/www/html -d reidmachine.party -d www.reidmachine.party Saving debug log to /var/log/letsencrypt/letsencrypt.log Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org ------------------------------------------------------------------------------- You have an existing certificate that contains a portion of the domains you requested (ref: /etc/letsencrypt/renewal/reidmachine.party.conf) It contains these names: reidmachine.party You requested these names for the new certificate: reidmachine.party, www.reidmachine.party. Do you want to expand and replace this existing certificate with the new certificate? ------------------------------------------------------------------------------- (E)xpand/(C)ancel: E Renewing an existing certificate Performing the following challenges: http-01 challenge for reidmachine.party http-01 challenge for www.reidmachine.party Using the webroot path /var/www/html for all unmatched domains. Waiting for verification... Cleaning up challenges Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/reidmachine.party/fullchain.pem. Your cert will expire on 2017-06-23. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le
Okay, so private and public key certificate file stuffs are here:
/etc/letsencrypt/keys/0001_key-certbot.pem
Certbot cron job
Certbot package will run a cron job to renew certificates before expiry. The certificates last for 90 days. To do a dry run of a certificate renewal:
$ sudo certbot renew --dry-run Saving debug log to /var/log/letsencrypt/letsencrypt.log ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/reidmachine.party.conf ------------------------------------------------------------------------------- Cert not due for renewal, but simulating renewal for dry run Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org Renewing an existing certificate Performing the following challenges: http-01 challenge for reidmachine.party http-01 challenge for www.reidmachine.party Waiting for verification... Cleaning up challenges Generating key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem ** DRY RUN: simulating 'certbot renew' close to cert expiry ** (The test certificates below have not been saved.) Congratulations, all renewals succeeded. The following certs have been renewed: /etc/letsencrypt/live/reidmachine.party/fullchain.pem (success) ** DRY RUN: simulating 'certbot renew' close to cert expiry ** (The test certificates above have not been saved.) IMPORTANT NOTES: - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.
Docker
At this point, we're ready to install docker, and then load up the new node with whatever containers we need.
Installing
Here is an official one-and-done bash script from Docker: https://git.charlesreid1.com/dotfiles/debian/src/master/get-docker.sh
Non-Sudo Docker Usage
Modify the non-root user to be able to run docker by adding them to the docker group:
groupadd docker usermod -a -G docker username
Docker Boats
Now you're all set up, and it's on to setting up your docker boats.
Once we've gone from "bare metal" to Docker, we're all set up to build a toolchain for whatever this node happens to be doing.
Next step is to build Docker boats - groups of containers - for different projects. See Docker/Boats.