As my main SSD is running low on estimated life remaining, I am attempting to containerize my projects so that they can be easily moved. The primary one is my Linux Apache MySQL PHP (LAMP) webserver that I use for lists and MythWeb.
For this project I will be using docker compose to put a nginx reverse proxy in front of Apache so that it can direct the direct the traffic, handle ssl encrpytion, and authentication. We will first be setting it up on some dev ports (9080/9443) so that we can test before replacing the existing servers.
Move docker data onto zfs
- This will keep the images and logs on zfs instead of my root drive
- It will also require any image to be downloaded again and rebuilt
- Stop dockerd
- sudo systemctl stop docker
- sudo systemctl stop docker.socket
- Move the docker data
- sudo mkdir /storage/containers/dockerd
- sudo rsync -avh --progress /var/lib/docker/ /storage/containers/dockerd
- sudo mv /var/lib/docker /var/lib/docker.old
- edit /etc/docker/daemon.json
{
"data-root": "/storage/containers/dockerd",
"storage-driver": "zfs"
}
- Restart dockerd
- sudo systemctl start docker.socket
- sudo systemctl start docker
General Setup
- Create a place to store all the files
- this should be on your zfs dataset
- sudo mkdir -p /storage/containers/webserver
- We will use this as the root directory for all of the below configs
- Create the needed subdirs
- cd /storage/containers/webserver
- sudo mkdir -p letsencrypt/etc letsencrypt/data letsencrypt/logs
- sudo mkdir -p nginx/www/html
- sudo mkdir -p lists/build lists/mysql lists/www/html/lists
First setup nginx
- Setup valid users for authentication
- sudo htpasswd -c nginx/www/htpasswd username
- Copy the existing letsencrypt certs
- sudo mkdir -p letsencrypt/etc/live/this.example.com
- sudo cp /etc/letsencrypt/live/this.example.com/* letsencrypt/etc/live/this.example.com/
- create a docker-compose.yml with the following contents:
# Begin docker-compose.ymlversion: '3.4'services:nginx:container-name: 'nginx-proxy'hostname: 'nginx-proxy'image: nginx:latestports:- "9080:80"- "9443:443"volume:- ./prod.conf:/etc/nginx/conf.d/default.conf- ./nginx/www:/www- ./letsencrypt/etc:/etc/letsencrypt- ./letsencrypt/data:/data/letsencrypt# End docker-compose.yml
- create a prod.conf with the following contents:
# Begin prod.confserver {listen 80;listen [::]:80;server_name this.example.com;location / {rewrite ^ https://$host:9443$request_uri? permanent;}# for cerbot challengelocation /.well-known/acme-challenge {allow all;root /data/letsencrypt;}}server {listen 443 ssl;listen [::]:443 ssl;http2 on;server_name this.example.com;ssl_certificate /etc/letsencrypt/live/this.example.com/full chain.pem;ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;auth_basic "Your Server Message";auth_basic_user_file /www/htpasswd;location / {root /www/html;}}# End prod.conf
- create a nginx/www/html/index.html that will link to our actual contents, here is my example:
<html><body><p><a href="lists/">Lists</a></p><p><a href="mythweb/">MythWeb</a></p></body></html>
- Now test the server
- sudo docker-compose up --build
- Visit your site in a browser
- Ctrl+C to stop the server
Setup MythWeb
- Add the mythweb section to docker-compose.yml so that it looks like:
# Begin docker-compose.ymlversion: '3.4'services:nginx:container-name: 'nginx-proxy'hostname: 'nginx-proxy'image: nginx:latestrestart: alwaysports:- "9080:80"- "9443:443"volume:- ./nginx/prod.conf:/etc/nginx/conf.d/default.conf- ./nginx/www:/www- ./letsencrypt/etc:/etc/letsencrypt- ./letsencrypt/data:/data/letsencryptmythweb:container-name: 'myth-http'hostname: 'myth-http'image: dheaps/mythbackend:mythwebrestart: alwaysports:- "7080:80"environment:- DATABASE_HOST=localhost- DATABASE_NAME=mythconverg- DATABASE_USER=mythtv- DATABASE_PASSWORD=YourSuperSecretPassword- TZ=America/New_Yorkvolumes:# This will have mysql connect over sockets instead ports- /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.sock# End docker-compose.yml
- Add the mythweb sections to prod.conf so that it looks like this:
# Begin nginx/prod.confserver {listen 80;listen [::]:80;server_name this.example.com;location / {rewrite ^ https://$host:9443$request_uri? permanent;}# for cerbot challengelocation /.well-known/acme-challenge {allow all;root /data/letsencrypt;}}server {listen 443 ssl;listen [::]:443 ssl;http2 on;server_name this.example.com;ssl_certificate /etc/letsencrypt/live/this.example.com/fullchain.pem;ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;auth_basic "Your Server Message";auth_basic_user_file /www/htpasswd;location / {root /www/html;}location /mythweb {# Use this to preserve port numberreturn 301 $scheme://$http_host/mythweb/;}location /mythweb/ {proxy_pass http://myth-http:80/mythweb/;proxy_buffering off;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-Proto $scheme;# using $http_host so that the links will include portproxy_set_header X-Forwarded-Host $http_host;proxy_set_header X-Forwarded-Port $server_port;}}# End nginx/prod.conf
- Now test the server
- sudo docker-compose up --build
- Visit your site in a browser
- Ctrl+C to stop the server
Setup MySQL and Apache
- Note: we are using MYSQL_HOST environment variable so that the PHP can easily switch what MySQL instance to connect to
- Create a directory to hold related files
- mkdir -p lists/build
- mkdir -p lists/mysql
- Copy your HTML/PHP files into lists/www/html/
- sudo mkdir -p lists/www/html/lists
- sudo cp /var/www/html/lists/* lists/www/html/lists/
- Add the MySQL and Apache sections to docker-compose.yml
# Begin docker-compose.ymlversion: '3.4'services:nginx:container-name: 'nginx-proxy'hostname: 'nginx-proxy'image: nginx:latestrestart: alwaysports:- "9080:80"- "9443:443"volume:- ./nginx/prod.conf:/etc/nginx/conf.d/default.conf- ./nginx/www:/www- ./letsencrypt/etc:/etc/letsencrypt- ./letsencrypt/data:/data/letsencryptmythweb:container-name: 'myth-http'hostname: 'myth-http'image: dheaps/mythbackend:mythwebrestart: alwaysports:- "7080:80"environment:- DATABASE_HOST=localhost- DATABASE_NAME=mythconverg- DATABASE_USER=mythtv- DATABASE_PASSWORD=YourSuperSecretPassword- TZ=America/New_Yorkvolumes:# This will have mysql connect over sockets instead ports- /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.socklists-www:container-name: 'lists-www'hostname: 'lists-www'build: './lists/build'restart: alwaysenvironment:- MYSQL_HOST=lists-mysqlvolumes:- ./lists/http.conf:/etc/apache2/httpd.conf- ./lists/www:/var/www# This is just where I keep my lists and isn't required- /home/user/Documents/lists:/home/user/Documents/listslists-mysql:container-name: 'lists-mysql'hostname: 'lists-mysql'image: mysql:latestrestart: alwaysports:- "4306:3306"environment:- MYSQL_ROOT_PASSWORD=AnotherSuperSecretPasswordvolumes:- ./lists/mysql:/var/lib/mysql# End docker-compose.yml
- Create lists/build/Dockerfile
# Begin lists/build/DockerfileFROM php:apacheRUN apt-get update && docker-php-ext-install mysqli pdo pdo_mysql# End lists/build/Dockerfile
- Create lists/http.conf
# Begin lists/http.conf<VirtualHost>ServerName this.example.comServerAdmin admin@this.example.comDocumentRoot /var/www/html</VirtualHost># End lists/http.conf
- Now test the server
- sudo docker-compose up --build
- While the test server is up load data into your MySQL instance
- Note: Don't use localhost or MySQL will ignore the port and use sockets
- sudo mysql -p -h 127.0.0.1 --port=4306
- Visit your site in a browser
- Ctrl+C to stop the server
Move to production
- Edit the configs
- in docker-compose.yml replace 9080 with 80 and 9443 with 443
- in nginx/prod.conf replace 9443 with 443
- Stop the normal apache and keep it from starting at boot
- sudo systemctl stop apache2
- sudo systemctl disable apache2
- Stop the normal certbot
- sudo systemctl disable certbot.timer
- Have docker compose start at startup
- /etc/systemd/system/docker-compose-webserver.service
# Begin /etc/systemd/system/docker-compose-webserver.service# Only include mysql.service if dependent on it for mythweb[Unit]Description=Docker Compose Webserver ServiceRequires=docker.serviceWants=mysql.serviceAfter=docker.service mysql.service[Service]Type=oneshotRemainAfterExit=yesWorkingDirectory=/storage/containers/webserverExecStart=/usr/bin/docker-compose up --build -dExecStop=/usr/bin/docker-compose downTimeoutStartSec=0[Install]WantedBy=multi-user.target# End /etc/systemd/system/docker-compose-webserver.service
- Load and start docker-compose-webserver
- sudo systemctl daemon-reload
- sudo systemctl enable docker-compose-webserver
- sudo systemctl start docker-compose-webserver
Setup the certificate renewal
- Unfortunately, the site needs to be live before we can test/setup certificate renewal, so make sure you did the steps above
- Remove the certs that we copied in, as certbot needs a blank folder or it will add -0001 to our hostname directory
- sudo rm -rf ./letsencrypt/etc/*
- Use staging (the test environment) to check the commands
- If the above worked then we can try a live renewal (which is rate limited)
- If the above worked then we schedule automatic renewal
- Create if does not exist /lib/systemd/system/cerbot.service
# Begin certbot.timer[Unit]Description=Run certbot twice daily[Timer]OnCalendar=*-*-* 00,12:00:00RandomizedDelaySec=43200Persistent=true[Install]WantedBy=timers.target# End certbot.timer
- Edit/create /lib/systemd/system/cerbot.service
# Begin certbot.service[Unit]Description=CertbotDocumentation=file:///usr/share/doc/python-certbot-doc/html/index.htmlDocumentation=letsencrypt.readthedocs.io/en/latest[Service]Type=oneshot# ExecStart=/usr/bin/certbot -q renewWorkingDirectory=/storage/containers/webserverExecStart=docker run --rm \-v ./letsencrypt/data:/data/letsencrypt \-v ./letsencrypt/etc:/etc/letsencrypt \-v ./letsencrypt/logs:/var/logs/letsencrypt \certbot/certbot \renew --quiet --webroot \--email youremail@domain.com --agree-tos --no-eff-email \--webroot-path=/data/letsencryptExecStartPost=docker exec nginx-proxy nginx -s reloadPrivateTmp=true# End certbot.service
- Enable the service
- sudo systemctl daemon-reload
- sudo systemctl enable certbot.timer
Update 2024-05-09: You need to restart nginx after getting a new certificate. I haven't figured out the best way to automate this.
Update 2024-05-10: Added the ExecStartPost in certbot.service to have nginx reload after renewing
Update 2024-05-26: Added `restart: always` so that the services will restore after they fail or docker fails.
Debug
- If you see errors like `Cannot create container for service`
- view all containers:
- docker ps -a
- you can remove the offending container with:
- docker rm <container-name>
Next Steps
- Put mythbackend and its MySQL instance in docker
Appendix
Sources
- Docker
- https://docs.docker.com/reference/cli/dockerd/#daemon-configuration-file
- https://tienbm90.medium.com/how-to-change-docker-root-data-directory-89a39be1a70b
- https://mrkandreev.name/snippets/how_to_move_docker_data_to_another_location/
- https://www.guguweb.com/2019/02/07/how-to-move-docker-data-directory-to-another-location-on-ubuntu/
- https://stackoverflow.com/questions/68634012/how-to-fix-docker-storage-driver-overlay2-problem
- https://docs.docker.com/storage/storagedriver/select-storage-driver/
- Apache/LetsEncrypt
- https://www.linkedin.com/pulse/use-apache-ssllets-encrypt-docker-leon-sczepansky/
- https://www.humankode.com/ssl/how-to-set-up-free-ssl-certificates-from-lets-encrypt-using-docker-and-nginx/
- https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/Simple-Apache-docker-compose-example-with-Dockers-httpd-image
- https://letsencrypt.org/docs/staging-environment/
- PHP
- https://doc4dev.com/en/create-a-web-site-php-apache-mysql-in-5-minutes-with-docker/
- https://thriveread.com/apache-php-with-docker-mysql-and-phpmyadmin/
- nginx
- https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/
- https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-to-setup-Nginx-reverse-proxy-servers-by-example
- https://stackoverflow.com/questions/41605137/how-to-remove-a-subdirectory-from-the-url-before-passing-to-a-script
- https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/
- https://www.hostwinds.com/tutorials/create-use-htpasswd
- MySQL
- MythTV
- Systemd