Tech Tips: March 2024

As my main SSD is running low on estimated life remaining, I am attempting to containerize my projects so that they can be easily moved. The primary one is my Linux Apache MySQL PHP (LAMP) webserver that I use for lists and MythWeb.

For this project I will be using docker compose to put a nginx reverse proxy in front of Apache so that it can direct the direct the traffic, handle ssl encrpytion, and authentication. We will first be setting it up on some dev ports (9080/9443) so that we can test before replacing the existing servers.

Move docker data onto zfs

This will keep the images and logs on zfs instead of my root drive
It will also require any image to be downloaded again and rebuilt
Stop dockerd

sudo systemctl stop docker
sudo systemctl stop docker.socket

Move the docker data

sudo mkdir /storage/containers/dockerd
sudo rsync -avh --progress /var/lib/docker/ /storage/containers/dockerd
sudo mv /var/lib/docker /var/lib/docker.old

edit /etc/docker/daemon.json

{
"data-root": "/storage/containers/dockerd",
"storage-driver": "zfs"
}

Restart dockerd

sudo systemctl start docker.socket
sudo systemctl start docker

General Setup

Create a place to store all the files

this should be on your zfs dataset
sudo mkdir -p /storage/containers/webserver

We will use this as the root directory for all of the below configs
Create the needed subdirs

cd /storage/containers/webserver
sudo mkdir -p letsencrypt/etc letsencrypt/data letsencrypt/logs
sudo mkdir -p nginx/www/html
sudo mkdir -p lists/build lists/mysql lists/www/html/lists

First setup nginx

Setup valid users for authentication

sudo htpasswd -c nginx/www/htpasswd username

Copy the existing letsencrypt certs

sudo mkdir -p letsencrypt/etc/live/this.example.com
sudo cp /etc/letsencrypt/live/this.example.com/* letsencrypt/etc/live/this.example.com/

create a docker-compose.yml with the following contents:

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
      ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
# End docker-compose.yml

create a prod.conf with the following contents:

# Begin prod.conf
server {
listen 80;
listen [::]:80;
server_name this.example.com;

location / {
      rewrite ^ https://$host:9443$request_uri? permanent;
}

# for cerbot challenge
location /.well-known/acme-challenge {
      allow all;
      root /data/letsencrypt;
}
}

server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name this.example.com;

ssl_certificate /etc/letsencrypt/live/this.example.com/full chain.pem;
ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;

auth_basic "Your Server Message";
auth_basic_user_file /www/htpasswd;

location / {
      root /www/html;
}
}
# End prod.conf

create a nginx/www/html/index.html that will link to our actual contents, here is my example:

<html>
<body>
<p>
<a href="lists/">Lists</a>
</p>
<p>
<a href="mythweb/">MythWeb</a>
</p>
</body>
</html>

Now test the server

sudo docker-compose up --build

Visit your site in a browser
Ctrl+C to stop the server

Setup MythWeb

Add the mythweb section to docker-compose.yml so that it looks like:

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
      ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./nginx/prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
mythweb:
      container-name: 'myth-http'
      hostname: 'myth-http'
      image: dheaps/mythbackend:mythweb
      ports:
          - "7080:80"
      environment:
          - DATABASE_HOST=localhost
          - DATABASE_NAME=mythconverg
          - DATABASE_USER=mythtv
          - DATABASE_PASSWORD=YourSuperSecretPassword
          - TZ=America/New_York
      volumes:
          # This will have mysql connect over sockets instead ports
           - /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.sock
# End docker-compose.yml

Add the mythweb sections to prod.conf so that it looks like this:

# Begin nginx/prod.conf
server {
listen 80;
listen [::]:80;
server_name this.example.com;

location / {
      rewrite ^ https://$host:9443$request_uri? permanent;
}

# for cerbot challenge
location /.well-known/acme-challenge {
      allow all;
      root /data/letsencrypt;
}
}

server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name this.example.com;

ssl_certificate /etc/letsencrypt/live/this.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;

auth_basic "Your Server Message";
auth_basic_user_file /www/htpasswd;

location / {
      root /www/html;
}

location /mythweb {
      # Use this to preserve port number
      return 301 $scheme://$http_host/mythweb/;
}
location /mythweb/ {
      proxy_pass http://myth-http:80/mythweb/;
      proxy_buffering off;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      # using $http_host so that the links will include port
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Port $server_port;
}
}
# End nginx/prod.conf

Now test the server

sudo docker-compose up --build

Visit your site in a browser
Ctrl+C to stop the server

Setup MySQL and Apache

Note: we are using MYSQL_HOST environment variable so that the PHP can easily switch what MySQL instance to connect to
Create a directory to hold related files

mkdir -p lists/build
mkdir -p lists/mysql

Copy your HTML/PHP files into lists/www/html/

sudo mkdir -p lists/www/html/lists
sudo cp /var/www/html/lists/* lists/www/html/lists/

Add the MySQL and Apache sections to docker-compose.yml

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
      ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./nginx/prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
mythweb:
      container-name: 'myth-http'
      hostname: 'myth-http'
      image: dheaps/mythbackend:mythweb
      ports:
          - "7080:80"
      environment:
          - DATABASE_HOST=localhost
          - DATABASE_NAME=mythconverg
          - DATABASE_USER=mythtv
          - DATABASE_PASSWORD=YourSuperSecretPassword
          - TZ=America/New_York
      volumes:
          # This will have mysql connect over sockets instead ports
          - /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.sock
lists-www:
      container-name: 'lists-www'
      hostname: 'lists-www'
      build: './lists/build'
      environment:
          - MYSQL_HOST=lists-mysql
      volumes:
          - ./lists/http.conf:/etc/apache2/httpd.conf
          - ./lists/www:/var/www
          # This is just where I keep my lists and isn't required
          - /home/user/Documents/lists:/home/user/Documents/lists
lists-mysql:
      container-name: 'lists-mysql'
      hostname: 'lists-mysql'
      image: mysql:latest
      ports:
          - "4306:3306"
      environment:
          - MYSQL_ROOT_PASSWORD=AnotherSuperSecretPassword
      volumes:
          - ./lists/mysql:/var/lib/mysql
# End docker-compose.yml

Create lists/build/Dockerfile

# Begin lists/build/Dockerfile
FROM php:apache
RUN apt-get update && docker-php-ext-install mysqli pdo pdo_mysql
# End lists/build/Dockerfile

Create lists/http.conf

# Begin lists/http.conf
<VirtualHost>
ServerName this.example.com
ServerAdmin admin@this.example.com
DocumentRoot /var/www/html
</VirtualHost>
# End lists/http.conf

Now test the server

sudo docker-compose up --build

While the test server is up load data into your MySQL instance

Note: Don't use localhost or MySQL will ignore the port and use sockets
sudo mysql -p -h 127.0.0.1 --port=4306

Visit your site in a browser
Ctrl+C to stop the server

Move to production

Edit the configs

in docker-compose.yml replace 9080 with 80 and 9443 with 443
in nginx/prod.conf replace 9443 with 443

Stop the normal apache and keep it from starting at boot

sudo systemctl stop apache2
sudo systemctl disable apache2

Stop the normal certbot

sudo systemctl disable certbot.timer

Have docker compose start at startup

/etc/systemd/system/docker-compose-webserver.service

# Begin /etc/systemd/system/docker-compose-webserver.service
# Only include mysql.service if dependent on it for mythweb
[Unit]
Description=Docker Compose Webserver Service
Requires=docker.service
Wants=mysql.service
After=docker.service mysql.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/storage/containers/webserver
ExecStart=/usr/bin/docker-compose up --build -d
ExecStop=/usr/bin/docker-compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target
# End /etc/systemd/system/docker-compose-webserver.service

Load and start docker-compose-webserver

sudo systemctl daemon-reload
sudo systemctl enable docker-compose-webserver
sudo systemctl start docker-compose-webserver

Setup the certificate renewal

Unfortunately, the site needs to be live before we can test/setup certificate renewal, so make sure you did the steps above
Remove the certs that we copied in, as certbot needs a blank folder or it will add -0001 to our hostname directory

sudo rm -rf ./letsencrypt/etc/*

Use staging (the test environment) to check the commands

sudo docker run -it --rm \

-v ./letsencrypt/data:/data/letsencrypt \

-v ./letsencrypt/etc:/etc/letsencrypt \

-v ./letsencrypt/logs:/var/logs/letsencrypt \

certbot/certbot \

certonly --webroot \

--register-unsafely-without-email --agree-tos \

--webroot-path=/data/letsencrypt \

--staging \

-d this.example.com

If the above worked then we can try a live renewal (which is rate limited)