Tech Tips

23 March 2024

Dockerize my LAMP webserver

As my main SSD is running low on estimated life remaining, I am attempting to containerize my projects so that they can be easily moved. The primary one is my Linux Apache MySQL PHP (LAMP) webserver that I use for lists and MythWeb.

For this project I will be using docker compose to put a nginx reverse proxy in front of Apache so that it can direct the direct the traffic, handle ssl encrpytion, and authentication. We will first be setting it up on some dev ports (9080/9443) so that we can test before replacing the existing servers.

Move docker data onto zfs

This will keep the images and logs on zfs instead of my root drive
It will also require any image to be downloaded again and rebuilt
Stop dockerd

sudo systemctl stop docker
sudo systemctl stop docker.socket

Move the docker data

sudo mkdir /storage/containers/dockerd
sudo rsync -avh --progress /var/lib/docker/ /storage/containers/dockerd
sudo mv /var/lib/docker /var/lib/docker.old

edit /etc/docker/daemon.json

{
"data-root": "/storage/containers/dockerd",
"storage-driver": "zfs"
}

Restart dockerd

sudo systemctl start docker.socket
sudo systemctl start docker

General Setup

Create a place to store all the files

this should be on your zfs dataset
sudo mkdir -p /storage/containers/webserver

We will use this as the root directory for all of the below configs
Create the needed subdirs

cd /storage/containers/webserver
sudo mkdir -p letsencrypt/etc letsencrypt/data letsencrypt/logs
sudo mkdir -p nginx/www/html
sudo mkdir -p lists/build lists/mysql lists/www/html/lists

First setup nginx

Setup valid users for authentication

sudo htpasswd -c nginx/www/htpasswd username

Copy the existing letsencrypt certs

sudo mkdir -p letsencrypt/etc/live/this.example.com
sudo cp /etc/letsencrypt/live/this.example.com/* letsencrypt/etc/live/this.example.com/

create a docker-compose.yml with the following contents:

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
      ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
# End docker-compose.yml

create a prod.conf with the following contents:

# Begin prod.conf
server {
listen 80;
listen [::]:80;
server_name this.example.com;

location / {
      rewrite ^ https://$host:9443$request_uri? permanent;
}

# for cerbot challenge
location /.well-known/acme-challenge {
      allow all;
      root /data/letsencrypt;
}
}

server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name this.example.com;

ssl_certificate /etc/letsencrypt/live/this.example.com/full chain.pem;
ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;

auth_basic "Your Server Message";
auth_basic_user_file /www/htpasswd;

location / {
      root /www/html;
}
}
# End prod.conf

create a nginx/www/html/index.html that will link to our actual contents, here is my example:

<html>
<body>
<p>
<a href="lists/">Lists</a>
</p>
<p>
<a href="mythweb/">MythWeb</a>
</p>
</body>
</html>

Now test the server

sudo docker-compose up --build

Visit your site in a browser
Ctrl+C to stop the server

Setup MythWeb

Add the mythweb section to docker-compose.yml so that it looks like:

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
  restart: always
ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./nginx/prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
mythweb:
      container-name: 'myth-http'
      hostname: 'myth-http'
      image: dheaps/mythbackend:mythweb
        restart: always
ports:
          - "7080:80"
      environment:
          - DATABASE_HOST=localhost
          - DATABASE_NAME=mythconverg
          - DATABASE_USER=mythtv
          - DATABASE_PASSWORD=YourSuperSecretPassword
          - TZ=America/New_York
      volumes:
          # This will have mysql connect over sockets instead ports
           - /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.sock
# End docker-compose.yml

Add the mythweb sections to prod.conf so that it looks like this:

# Begin nginx/prod.conf
server {
listen 80;
listen [::]:80;
server_name this.example.com;

location / {
      rewrite ^ https://$host:9443$request_uri? permanent;
}

# for cerbot challenge
location /.well-known/acme-challenge {
      allow all;
      root /data/letsencrypt;
}
}

server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name this.example.com;

ssl_certificate /etc/letsencrypt/live/this.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/this.example.com/privkey.pem;

auth_basic "Your Server Message";
auth_basic_user_file /www/htpasswd;

location / {
      root /www/html;
}

location /mythweb {
      # Use this to preserve port number
      return 301 $scheme://$http_host/mythweb/;
}
location /mythweb/ {
      proxy_pass http://myth-http:80/mythweb/;
      proxy_buffering off;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      # using $http_host so that the links will include port
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Port $server_port;
}
}
# End nginx/prod.conf

Now test the server

sudo docker-compose up --build

Visit your site in a browser
Ctrl+C to stop the server

Setup MySQL and Apache

Note: we are using MYSQL_HOST environment variable so that the PHP can easily switch what MySQL instance to connect to
Create a directory to hold related files

mkdir -p lists/build
mkdir -p lists/mysql

Copy your HTML/PHP files into lists/www/html/

sudo mkdir -p lists/www/html/lists
sudo cp /var/www/html/lists/* lists/www/html/lists/

Add the MySQL and Apache sections to docker-compose.yml

# Begin docker-compose.yml
version: '3.4'

services:
nginx:
      container-name: 'nginx-proxy'
      hostname: 'nginx-proxy'
      image: nginx:latest
  restart: always
ports:
          - "9080:80"
          - "9443:443"
      volume:
          - ./nginx/prod.conf:/etc/nginx/conf.d/default.conf
          - ./nginx/www:/www
          - ./letsencrypt/etc:/etc/letsencrypt
        - ./letsencrypt/data:/data/letsencrypt
mythweb:
      container-name: 'myth-http'
      hostname: 'myth-http'
      image: dheaps/mythbackend:mythweb
  restart: always
ports:
          - "7080:80"
      environment:
          - DATABASE_HOST=localhost
          - DATABASE_NAME=mythconverg
          - DATABASE_USER=mythtv
          - DATABASE_PASSWORD=YourSuperSecretPassword
          - TZ=America/New_York
      volumes:
          # This will have mysql connect over sockets instead ports
          - /var/run/mysqld/mysql.sock:/var/run/mysqld/mysql.sock
lists-www:
      container-name: 'lists-www'
      hostname: 'lists-www'
      build: './lists/build'
        restart: always
environment:
          - MYSQL_HOST=lists-mysql
      volumes:
          - ./lists/http.conf:/etc/apache2/httpd.conf
          - ./lists/www:/var/www
          # This is just where I keep my lists and isn't required
          - /home/user/Documents/lists:/home/user/Documents/lists
lists-mysql:
      container-name: 'lists-mysql'
      hostname: 'lists-mysql'
      image: mysql:latest
        restart: always
ports:
          - "4306:3306"
      environment:
          - MYSQL_ROOT_PASSWORD=AnotherSuperSecretPassword
      volumes:
          - ./lists/mysql:/var/lib/mysql
# End docker-compose.yml

Create lists/build/Dockerfile

# Begin lists/build/Dockerfile
FROM php:apache
RUN apt-get update && docker-php-ext-install mysqli pdo pdo_mysql
# End lists/build/Dockerfile

Create lists/http.conf

# Begin lists/http.conf
<VirtualHost>
ServerName this.example.com
ServerAdmin admin@this.example.com
DocumentRoot /var/www/html
</VirtualHost>
# End lists/http.conf

Now test the server

sudo docker-compose up --build

While the test server is up load data into your MySQL instance

Note: Don't use localhost or MySQL will ignore the port and use sockets
sudo mysql -p -h 127.0.0.1 --port=4306

Visit your site in a browser
Ctrl+C to stop the server

Move to production

Edit the configs

in docker-compose.yml replace 9080 with 80 and 9443 with 443
in nginx/prod.conf replace 9443 with 443

Stop the normal apache and keep it from starting at boot

sudo systemctl stop apache2
sudo systemctl disable apache2

Stop the normal certbot

sudo systemctl disable certbot.timer

Have docker compose start at startup

/etc/systemd/system/docker-compose-webserver.service

# Begin /etc/systemd/system/docker-compose-webserver.service
# Only include mysql.service if dependent on it for mythweb
[Unit]
Description=Docker Compose Webserver Service
Requires=docker.service
Wants=mysql.service
After=docker.service mysql.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/storage/containers/webserver
ExecStart=/usr/bin/docker-compose up --build -d
ExecStop=/usr/bin/docker-compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target
# End /etc/systemd/system/docker-compose-webserver.service

Load and start docker-compose-webserver

sudo systemctl daemon-reload
sudo systemctl enable docker-compose-webserver
sudo systemctl start docker-compose-webserver

Setup the certificate renewal

Unfortunately, the site needs to be live before we can test/setup certificate renewal, so make sure you did the steps above
Remove the certs that we copied in, as certbot needs a blank folder or it will add -0001 to our hostname directory

sudo rm -rf ./letsencrypt/etc/*

Use staging (the test environment) to check the commands

sudo docker run -it --rm \

-v ./letsencrypt/data:/data/letsencrypt \

-v ./letsencrypt/etc:/etc/letsencrypt \

-v ./letsencrypt/logs:/var/logs/letsencrypt \

certbot/certbot \

certonly --webroot \

--register-unsafely-without-email --agree-tos \

--webroot-path=/data/letsencrypt \

--staging \

-d this.example.com

If the above worked then we can try a live renewal (which is rate limited)

sudo docker run -it --rm \

-v ./letsencrypt/data:/data/letsencrypt \

-v ./letsencrypt/etc:/etc/letsencrypt \

-v ./letsencrypt/logs:/var/logs/letsencrypt \

certbot/certbot \

certonly --webroot \

--email youremail@domain.com --agree-tos --no-eff-email \

--webroot-path=/data/letsencrypt \

-d this.example.com

If the above worked then we schedule automatic renewal
Create if does not exist /lib/systemd/system/cerbot.service

# Begin certbot.timer
[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target
# End certbot.timer

Edit/create /lib/systemd/system/cerbot.service

# Begin certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=letsencrypt.readthedocs.io/en/latest

[Service]
Type=oneshot
# ExecStart=/usr/bin/certbot -q renew
WorkingDirectory=/storage/containers/webserver
ExecStart=docker run --rm \
-v ./letsencrypt/data:/data/letsencrypt \
-v ./letsencrypt/etc:/etc/letsencrypt \
-v ./letsencrypt/logs:/var/logs/letsencrypt \
certbot/certbot \
renew --quiet --webroot \
--email youremail@domain.com --agree-tos --no-eff-email \
--webroot-path=/data/letsencrypt
ExecStartPost=docker exec nginx-proxy nginx -s reload
PrivateTmp=true
# End certbot.service

Enable the service

sudo systemctl daemon-reload
sudo systemctl enable certbot.timer

Update 2024-05-09: You need to restart nginx after getting a new certificate. I haven't figured out the best way to automate this.

Update 2024-05-10: Added the ExecStartPost in certbot.service to have nginx reload after renewing

Update 2024-05-26: Added `restart: always` so that the services will restore after they fail or docker fails.

Debug

If you see errors like `Cannot create container for service`

view all containers:

docker ps -a

you can remove the offending container with:

docker rm <container-name>

Next Steps

Put mythbackend and its MySQL instance in docker

Appendix

Sources

Docker

Apache/LetsEncrypt

nginx

MySQL

https://earthly.dev/blog/docker-mysql/

MythTV

https://hub.docker.com/r/dheaps/mythbackend

Systemd

02 February 2024

New VNC client for ChromeOS

As RealVNC discontinued their ChromeOS version and it was giving me issues with disconnections, I decided to look for a replacement.

The search

xtightvncviewer

Bad connection dialog
Does not handle ChromeOS scaling properly
No client window scaling without changing host resolution

ssvnc

Bad connection dialog
Does not handle ChromeOS scaling properly
No client window scaling without changing host resolution
Puts 2 icons on the dock
Supports ssh/ssl encryption

tigervnc-viewer

Acceptable connection dialog
Does not handle ChromeOS scaling properly
No client window scaling without changing host resolution
Support TLS encryption

vinagre

Acceptable connection dialog
Handles ChromeOS scaling properly
Suppports client window scaling without changing host resolution
Dock icon didn't load properly
Touchpad scrolling does not work
No longer maintained, superseded by Gnome Connections

Gnome Connections

Slick looking connections page
Handles ChromeOS scaling properly
Supports client window scaling without changing host resolution
Supports TLS encryption
Touchpad scrolling does not work
Does not remember/resize window when connecting

After my search I decided to go with TigerVNC viewer, but will keep Gnome Connections installed as it may eventually overtake it. Below is how I installed each

Installing TigerVNC viewer

Launch terminal

sudo apt install tigervnc-viewer

Configure it to scale properly

First determine your Chromebooks scaling

Settings -> Displays -> Display size

Test to make sure it is what you like, where .8 == 80% from above

/usr/bin/sommelier -X --scale=.8 /usr/bin/xtigervncviewer

Edit xtightvncviewer.desktop

mkdir ${HOME}/.local/share/applications/
cp /usr/share/applications/xtigervncviewer.desktop ${HOME}/.local/share/applications/
vi ${HOME}/.local/share/applications/xtigervncviewer.desktop
find "Exec"
and set it to the command that you tested

Launch the App

Search Key -> TigerVNC

Connect

192.168.1.XXX:1

Installing Gnome Connections

Resize your linux storage size

Settings -> Advanced -> Linux development environment -> Disk size "Change"
I set it to 16 GB

Launch terminal

Make sure apt is up to date

sudo apt update
sudo apt upgrade

Install flatpak

sudo apt install flatpak
flatpak --user remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
Restart the linux container by right clicking on your terminal icon and select "Shut down Linux"

Install and start Gnome Connections

flatpak install flathub org.gnome.Connections
flatpak run org.gnome.Connections

Add your VNC server

192.168.1.XXX:5901
Make sure to select VNC

Appendix

Sources:

26 January 2024

Miscellaneous Minecraft Matters

I wanted to host a Minecraft server for me and my daughter to be able to play on. After some research, using a docker container surfaced as the easiest way to do this. Also I wanted to use Steam Link to be able to play, so I needed to add a shortcut to Minecraft in Steam.

Below are the steps that I used to accomplish this

Minecraft Bedrock Docker Server

https://github.com/itzg/docker-minecraft-bedrock-server
find your players XUID

I did this by starting the server connecting and looking at the server output

create a docker-compose.yml file where OPS has your players XUIDs in a comma separated list and ALLOW_LIST_USERS has the player names and XUIDs that you want to be able to login

version: '3.4'

services:
bds:
image: itzg/minecraft-bedrock-server
environment:
EULA: "TRUE"
GAMEMODE: creative
DIFFICULTY: peaceful
SERVER_NAME: "Our World"
OPS: "1234,5678"
ALLOW_CHEATS: "true"
ALLOW_LIST: "true"
ALLOW_LIST_USERS: "player1:1234,player with spaces:5678"
ports:
- "19132:19132/udp"
volumes:
- /storage/containers/minecraft/world1:/data
stdin_open: true
tty: true

start the container

docker-compose up

A permissions.json file will be created giving the specified players ops powers
Note: even though your server is local the Playstation/Xbox/Switch version will not be able to connect without a PS Plus/Xbox Live/Nintendo Online subscription

https://bugs.mojang.com/browse/BDS-1634

Adding a shortcut to Minecraft in Steam

Find out where Minecraft was stored

Paste the following into an explorer window:

%LocalAppData%\Packages\

Find the folder like:

Microsoft.MinecraftUWP_<seemingly_random_letters_and_numbers>

The seemingly random letters and numbers are the app id, we will need them for later

In Steam go to your Library and click "ADD A GAME" and then "Add a Non-Steam Game..."
Navigate to C:\Windows and select explorer.exe
You will see a new entry in your library called explorer
Right click on it -> Properties
Choose an appropriate icon
Rename it
Click "SET LAUNCH OPTIONS"
type/paste in the following:

shell:appsFolder\<your-app-id>!App

Click "OK"
Click "CLOSE"
You should now be able to launch Minecraft from Steam

Appendix

Sources:

https://linustechtips.com/topic/547779-add-universal-apps-and-purchased-windows-store-games-to-steam-library-and-how-to-pin-steam-games-to-windows-taskbar-and-start-menu/

04 January 2024

Changing a zpool from ashift=9 to ashift=12

I wanted the additional write speed on my nas drives that come from aligning the ashift value with physical sector size of my hard drives (ashift=9 is 512 bytes and ashift=12 is 4KB). Unfortunately, you cannot change ashift on an existing zpool, so you will have to backup the data, destroy the pool, recreate it, and then restore the data.

Prereq

pv (to monitor the process/speed)

sudo apt-get install pv

encrypted zfs data with the "wrong" ashift value that you want to migrate

Process to move

Stop any process that writes to your storage that you want to move
Setup temporary storage location

sudo zpool create external-storage mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXX /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXX
sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase external-storage/encrypted

Create a snapshot

sudo zfs snapshot storage/encrypted@migrate-20231229
sudo zfs list -t snapshot

Copy snapshot over

sudo bash -c 'zfs send storage/encrypted@migrate-20231229 | pv | zfs recv external-storage/encrypted/backup'

Ensure that all files have been backed up
Unmount the datasets

sudo zfs unmount storage/encrypted

Destroy the old zpool

sudo zpool destroy storage

Create the new zpool

sudo zpool create storage mirror /dev/disk/by-id/ata-WDC_WD60EFZX-68B3FN0_WD-XXXXXXX ata-WDC_WD60EFZX-68B3FN0_WD-XXXXXXX

Ensure is setup with the correct ashift value

sudo zdb -C storage | grep ashift

Create a temporary file to contain your passphrase because since zfs recv is using stdin to pull in the data it cannot prompt for it

echo "super-secret" > /home/example/passphrase.txt

Copy snapshot back

sudo bash -c 'zfs send external-storage/encrypted/backup@migrate-20231229 | pv | zfs recv -o encryption=aes-256-gcm -o keylocation=file:///home/example/passphrase.txt -o keyformat=passphrase storage/encrypted'

Change from a file to prompt for password

sudo zfs change-key -o keylocation=prompt storage/encrypted

Remove the temp passphrase

rm /home/example/passphrase.txt

Check that all your files are back in place
Now if you want you can destroy the backup or export it and keep the backup

sudo zpool destroy external-storage
OR
sudo zpool export external-storage

Appendix

If you see an error like:

cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
That means that you cannot copy to the encrypted dataset. What I did to get around this was to instead copy to a child of the encrypted dataset.

Sources:

01 January 2024

Migrating from eCryptFS to native zfs encryption

I wanted to move from eCryptFS on top of a zfs dataset to a more standard and speedier encryption approach which is native zfs encryption. Here is the process that I went through.

Process

Ensure your backups are up to date!
Upgrade the zpool

ensure you are on a recent version of zfs and not zfs-fuse (see previous post)
sudo zpool upgrade storage

Create the destination dataset

sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase storage/new-encrypt

Set/change the mount point (optional)

sudo zfs set mountpoint=/storage/new-encrypt storage/new-encrypt

Move the files over

sudo rsync -avh --progress --remove-source-files /storage/encrypted/* /storage/new-encrypt/
-z / --compress is not needed and would slow down a local transfer

Remove the left over directories

sudo find /storage/encrypted/ -type d -empty -delete

Verify no files are left:

ls -al /storage/encrypted
if any files exist then repeat the rsync

Unmount the encryptfs

sudo umount /storage/encrypted

Remove/comment the entry from /etc/fstab

sudo vi /etc/fstab

Unmount the zfs dataset

sudo zfs unmount storage/.encrypted

Test destroying the zfs dataset

sudo zfs destroy -n storage/.encrypted

Destroy the zfs dataset

sudo zfs destroy storage/.encrypted

Change the name of new-encrypt

sudo zfs rename storage/new-encrypt storage/encrypted

Update mountpoint (if required)

sudo zfs set mountpoint=/storage/encrypted storage/encrypted

If everything works then the new zfs native encrypted dataset slots right into where the old one was and all your samba shares should be fine.

Appendix

Sources:

https://rucore.net/en/luks-ecryptfs-or-zfs-encryption-choosing-a-way-to-protect-your-data-in-linux/

13 December 2023

Adventures with ZFS

As part of plan to improve my backup strategy, I was testing out different zfs configurations to see what might be the best option.

The Problem

I was having trouble testing out using native zfs encryption

sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt
invalid property 'encryption'

Checking zfs version

So first I checked the zfs version:

sudo modinfo zfs | grep version

version: 0.8.3-1ubuntu12.15

zpool version

returned an error, which I found weird

zpool upgrade -v

returned a maximum version of 23
expected a maximum version of 28

Did some googling and found out that I may have the older zfs-fuse installed
dpkg -s zfs-fuse

confirmed my suspicion and told me that I had 0.7.0 installed

Replace zfs-fuse with zfsutils-linux

* Unmount all zfs datasets*

I didn't do this step, but you definitely should to prevent data errors
sudo zfs unmount <zpool>/<dataset>

sudo apt remove zfs-fuse
sudo apt install zfsutils-linux
Here I rebooted the system
sudo zpool status

no zpools available

sudo zpool import -a

listed my zpool with its last use

sudo zpool import -f <zpool>
sudo zpool status

now correctly showed my zpool

Test setup

I am using 2 disks that have a raw speed of 150 MB/s (benchmarked using dd on a single disk)
Creating the test pool

sudo zpool create test-pool mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>
or
sudo zpool create test-pool -o ashift=9 mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>

Creating test dataset

sudo zfs create test-pool/test
sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt

Disable caching

sudo zfs set primarycache=none test-pool/test
sudo zfs set secondarycache=none test-pool/test

10 back-to-back copies of a 5GB file and waiting for the numbers to stabilize

sudo rsync --progress Downloads/Win11_22H2_English_x64v1.iso /test-pool/test/

Deleting the zpool when done testing

sudo zpool destroy test-pool

Results of the upgrade and test

The good news is that I am seeing a 2-3X speed up on my existing zpool setup with ashift=9 from doing the upgrade. It also appears that native encryption has a minimal impact (as long as your processor has AES-NI). Here are the numbers that I was seeing

zfs-fuse 0.7.0:

ashift=9, no encryption: 35-45 MB/s
ashift=12, no encryption: 70-72 MB/s

zfsutils-linux 0.8.3:

ashift=9, no encryption: 108-117 MB/s
ashift=9, native encryption: 105-117 MB/s
ashift=12, no encryption: 134-141 MB/s
ashift=12, native encryption: 133-139 MB/s

Appendix

Article on setting up native zfs encryption:

https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/

checking zfs version:

zfs-fuse being super outdated:

Importing missing zpool:

https://serverfault.com/questions/585228/zfs-import-unable-to-find-any-pools

More about ashift:

rsync stats:

https://unix.stackexchange.com/questions/65077/is-it-possible-to-see-cp-speed-and-percent-copied

09 December 2023

Updating Crucial MX500 Firmware in Linux

On my Crucial MX500, I was noticing a high level of write amplification, which is when you tell it to write 1GB of data, but it actually uses 10GB of writes to the nand flash. To try to fix this, I decided to see if a firmware update would help with this.

Identifying the Problem

Substitute /dev/sdX with you drive

Get the smart attributes

sudo smartctl -A /dev/sdX

write down the values for 247 and 248, I will refer to thus as A
wait a few days and repeat steps 1 and 2, I will refer to this as B
Now lets calculate

247C =247B - 247A
248C = 248B - 248A
(247C + 248C) / 247C

I was seeing values ranging from 10-100, when I believe the typical range should be 1-2.5

Performing the Update

Caution: Before doing any of this be sure that you have up to date backups!

Substitute /dev/sdX with your drive

Use smartctl to check what firmware version you currently have installed so you can download the correct version

sudo smartctl -i /dev/sdX
Example line: Firmware Version: M3CR020

Download the correct firmware for your device:

https://www.crucial.com/support/ssd-support/mx500-support

Mount the iso

sudo mkdir /mnt/iso
sudo mount -o loop,ro MX500_M3CR023_update.iso /mnt/iso

Create a directory to extract the files to

mkdir mx500
cd mx500

Do the extraction

gzip -dc /mnt/iso/boot/corepure64.gz | cpio -idm

List the drives

sudo ./sbin/msecli -L

Perform the Update

sudo ./sbin/msecli -U -v -i ./opt/firmware/ -n /dev/sdX

Conclusion

It seemed to help but has not completely resolved the problem

Appendix

Sources:

https://medium.com/@vdboor/upgrading-crucial-firmware-on-linux-76056254539

Others experiencing a similar write amplification issue:

Guide for calculating Write Amplification:

(PDF) https://www.micron.com/-/media/client/global/documents/products/technical-note/solid-state-storage/tnfd22_client_ssd_smart_attributes.pdf