Tech Tips: December 2023

13 December 2023

Adventures with ZFS

As part of plan to improve my backup strategy, I was testing out different zfs configurations to see what might be the best option.

The Problem

I was having trouble testing out using native zfs encryption

sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt
invalid property 'encryption'

Checking zfs version

So first I checked the zfs version:

sudo modinfo zfs | grep version

version: 0.8.3-1ubuntu12.15

zpool version

returned an error, which I found weird

zpool upgrade -v

returned a maximum version of 23
expected a maximum version of 28

Did some googling and found out that I may have the older zfs-fuse installed
dpkg -s zfs-fuse

confirmed my suspicion and told me that I had 0.7.0 installed

Replace zfs-fuse with zfsutils-linux

* Unmount all zfs datasets*

I didn't do this step, but you definitely should to prevent data errors
sudo zfs unmount <zpool>/<dataset>

sudo apt remove zfs-fuse
sudo apt install zfsutils-linux
Here I rebooted the system
sudo zpool status

no zpools available

sudo zpool import -a

listed my zpool with its last use

sudo zpool import -f <zpool>
sudo zpool status

now correctly showed my zpool

Test setup

I am using 2 disks that have a raw speed of 150 MB/s (benchmarked using dd on a single disk)
Creating the test pool

sudo zpool create test-pool mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>
or
sudo zpool create test-pool -o ashift=9 mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>

Creating test dataset

sudo zfs create test-pool/test
sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt

Disable caching

sudo zfs set primarycache=none test-pool/test
sudo zfs set secondarycache=none test-pool/test

10 back-to-back copies of a 5GB file and waiting for the numbers to stabilize

sudo rsync --progress Downloads/Win11_22H2_English_x64v1.iso /test-pool/test/

Deleting the zpool when done testing

sudo zpool destroy test-pool

Results of the upgrade and test

The good news is that I am seeing a 2-3X speed up on my existing zpool setup with ashift=9 from doing the upgrade. It also appears that native encryption has a minimal impact (as long as your processor has AES-NI). Here are the numbers that I was seeing

zfs-fuse 0.7.0:

ashift=9, no encryption: 35-45 MB/s
ashift=12, no encryption: 70-72 MB/s

zfsutils-linux 0.8.3:

ashift=9, no encryption: 108-117 MB/s
ashift=9, native encryption: 105-117 MB/s
ashift=12, no encryption: 134-141 MB/s
ashift=12, native encryption: 133-139 MB/s

Appendix

Article on setting up native zfs encryption:

https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/

checking zfs version:

zfs-fuse being super outdated:

Importing missing zpool:

https://serverfault.com/questions/585228/zfs-import-unable-to-find-any-pools

More about ashift:

rsync stats:

https://unix.stackexchange.com/questions/65077/is-it-possible-to-see-cp-speed-and-percent-copied

09 December 2023

Updating Crucial MX500 Firmware in Linux

On my Crucial MX500, I was noticing a high level of write amplification, which is when you tell it to write 1GB of data, but it actually uses 10GB of writes to the nand flash. To try to fix this, I decided to see if a firmware update would help with this.

Identifying the Problem

Substitute /dev/sdX with you drive

Get the smart attributes

sudo smartctl -A /dev/sdX

write down the values for 247 and 248, I will refer to thus as A
wait a few days and repeat steps 1 and 2, I will refer to this as B
Now lets calculate

247C =247B - 247A
248C = 248B - 248A
(247C + 248C) / 247C

I was seeing values ranging from 10-100, when I believe the typical range should be 1-2.5

Performing the Update

Caution: Before doing any of this be sure that you have up to date backups!

Substitute /dev/sdX with your drive

Use smartctl to check what firmware version you currently have installed so you can download the correct version

sudo smartctl -i /dev/sdX
Example line: Firmware Version: M3CR020

Download the correct firmware for your device:

https://www.crucial.com/support/ssd-support/mx500-support

Mount the iso

sudo mkdir /mnt/iso
sudo mount -o loop,ro MX500_M3CR023_update.iso /mnt/iso

Create a directory to extract the files to

mkdir mx500
cd mx500

Do the extraction

gzip -dc /mnt/iso/boot/corepure64.gz | cpio -idm

List the drives

sudo ./sbin/msecli -L

Perform the Update

sudo ./sbin/msecli -U -v -i ./opt/firmware/ -n /dev/sdX

Conclusion

It seemed to help but has not completely resolved the problem

Appendix

Sources:

https://medium.com/@vdboor/upgrading-crucial-firmware-on-linux-76056254539

Others experiencing a similar write amplification issue:

Guide for calculating Write Amplification:

(PDF) https://www.micron.com/-/media/client/global/documents/products/technical-note/solid-state-storage/tnfd22_client_ssd_smart_attributes.pdf

08 December 2023

Migrating from Unifi USG-3P to UDM Pro

Decision

As my USG-3P was getting on in age and no longer getting updates, I have been shopping around for a replacement. Below were the main competitors for me.

UXG-Lite:

Price: $129 (plus tax and shipping)
WiFi: None
Unifi controller: No
CPU: Two A53 cores at 1 GHz
Memory: 1 GB of DDR3L
Storage: N/A
IPS/IDS max throughput: claimed 1Gbps (subject to third party verification)
Unifi Protect: No support

UDR:

Price: $199 (plus tax and shipping)
WiFi: 6 (however, this was to be deployed in my basement so not much help)
Unifi controller: Yes
CPU: Two A53 cores at 1.35 GHz
Memory: 2 GB of ?DDR3L?
Storage: 128 GB SSD
IPS/IDS max throughput: ~700Mbps
Unifi Protect: would need a SD card

UDM Pro:

Price: $379 --Black Friday $279-- (plus tax and shipping)
WiFi: No
Unifi controller: Yes
CPU: Four A57 cores at 1.7 GHz
Memory: 4 GB of DDR4
Storage: 16 GB eMMC
IPS/IDS max throughput: ~3.5Gbps
Unifi Protect: Hard drive slot
Bonus: LAN + WAN SFP+ ports for 10Gbps networking

To me the extra money was worth it to get the UDM Pro for the Black Friday price of $279, but I would probably not for $379. It just seems like a much more capable product that provides the option to expand later (>1Gbps networking, Unifi Protect).

Initial Impressions

Positives:

Shipping box contained lots of air pillows
High quality foam protecting the device in device packaging
Build quality is superb
Slide out foam screw holder was a nice organizational touch

Negatives:

Single use plastics used to wrap:

UDM Pro itself
Rack mount ears
Instructions, really????

Installation

Background:

This guide was written using Unifi Network 8.0.7
USG-3P network address is 192.168.1.1
Unifi controller is hosted at https://192.168.1.2:8443

Requires:

Internet connection
Laptop with an ethernet port or a PC that can be hardwired
2x ethernet cords (Only 1 is needed if you have an already hardwired PC)

Here are the steps that I used:

Create a backup from your current Unifi controller on a laptop

Navigate to https://192.168.1.2:8443
Settings -> System -> Backups
Click on `Download`
Select number of days (I chose 7)
Click `Download`

Connect a LAN port on your current network to the WAN port on UDM Pro

This is to provide it with internet access

Power on the UDM Pro
Allow it to update (this took several minutes)
Connect a laptop to the LAN port on the UDM Pro

May be helpful to disable WiFi on laptop

Setup Wizard on UDM Pro

Navigate to https://unifi/ (for me https://192.168.0.1 also would have worked)
Login with your Unifi account
Do NOT restore from Backup, skip this step
Finish the setup Wizard

Update the Network Application

You want it to be >= Unifi controller Network version

Restore the backup

Network -> Settings -> System -> Backups
Click on `Restore`
Select the backup you created above
Click on `Restore`
UDM Pro will restart

UDM Pro web UI will become unresponsive

At this point I used the touch screen to reboot the UDM Pro
However, this could be unnecessary and may possibly be resolved by forcing the laptop to get a new DHCP address

Ensure your restore happened correctly

Navigate to https://192.168.1.1
Check that the network settings are correct and that your access points are there (but they won't be connected)

Swap out the USG-3P for the UDM Pro
Migrate the Access Points from Unifi controller to UDM Pro

Navigate to https://192.168.1.2:8443
Settings -> General
Click on `Export Site` on the bottom
You can save the export file, but we won't be using it
Click continue on the `Export Site` dialog
Click continue on the `Migrate Site` dialog (no action needed)
Type in the IP address of the UDM Pro (192.168.1.1)
Select the Access Points to migrate
Click `Migrate Devices`
Check in another tab/window that they migrated to the UDM Pro
Click on `Remove Devices`

Done!

All in all, it took me about 1.5 hours, but this included unboxing, attaching the rack mount ears, mounting in the rack, and some research. The good news is that the network downtime was less than 5 minutes!

Overall Impressions

Positives:

The migration went smoothly, once I pieced together what needed done
Network performance is great
IPS has already started blocking network scans

Negatives:

Will not fit in a 12" rack that is flush mounted

AC plug is very far to one side and almost didn't have enough clearance for the 2x4 supporting my network rack, wish it was more centered
Possibly could be resolved by a 90 degree power connector

There doesn't seem to be a way to manage the screen

It will go into a screensaver mode during the day and turn off at night
However, I cannot find settings to change the times for this behavior
Update 2024-01-04:

To change this you have to set the local account as "Super Admin" on https://unifi.ui.com -> UDM Pro -> OS Settings (At the top looks like a UDM Pro with a gear icon on the bottom right) -> Admins & Users
Then you can update it https://192.168.1.1/console-settings
I set night mode to start at 10:01 PM and end at 10:00 PM thus keeping the screen off for most of the day

Other thoughts:

Lack of detailed official documentation on migration process
Topology is wrong because I have a non-Unifi switch and there isn't a way to manually fix

Research that I found: