Tech Tips: 2023

13 December 2023

Adventures with ZFS

As part of plan to improve my backup strategy, I was testing out different zfs configurations to see what might be the best option.

The Problem

I was having trouble testing out using native zfs encryption

sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt
invalid property 'encryption'

Checking zfs version

So first I checked the zfs version:

sudo modinfo zfs | grep version

version: 0.8.3-1ubuntu12.15

zpool version

returned an error, which I found weird

zpool upgrade -v

returned a maximum version of 23
expected a maximum version of 28

Did some googling and found out that I may have the older zfs-fuse installed
dpkg -s zfs-fuse

confirmed my suspicion and told me that I had 0.7.0 installed

Replace zfs-fuse with zfsutils-linux

* Unmount all zfs datasets*

I didn't do this step, but you definitely should to prevent data errors
sudo zfs unmount <zpool>/<dataset>

sudo apt remove zfs-fuse
sudo apt install zfsutils-linux
Here I rebooted the system
sudo zpool status

no zpools available

sudo zpool import -a

listed my zpool with its last use

sudo zpool import -f <zpool>
sudo zpool status

now correctly showed my zpool

Test setup

I am using 2 disks that have a raw speed of 150 MB/s (benchmarked using dd on a single disk)
Creating the test pool

sudo zpool create test-pool mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>
or
sudo zpool create test-pool -o ashift=9 mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>

Creating test dataset

sudo zfs create test-pool/test
sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt

Disable caching

sudo zfs set primarycache=none test-pool/test
sudo zfs set secondarycache=none test-pool/test

10 back-to-back copies of a 5GB file and waiting for the numbers to stabilize

sudo rsync --progress Downloads/Win11_22H2_English_x64v1.iso /test-pool/test/

Deleting the zpool when done testing

sudo zpool destroy test-pool

Results of the upgrade and test

The good news is that I am seeing a 2-3X speed up on my existing zpool setup with ashift=9 from doing the upgrade. It also appears that native encryption has a minimal impact (as long as your processor has AES-NI). Here are the numbers that I was seeing

zfs-fuse 0.7.0:

ashift=9, no encryption: 35-45 MB/s
ashift=12, no encryption: 70-72 MB/s

zfsutils-linux 0.8.3:

ashift=9, no encryption: 108-117 MB/s
ashift=9, native encryption: 105-117 MB/s
ashift=12, no encryption: 134-141 MB/s
ashift=12, native encryption: 133-139 MB/s

Appendix

Article on setting up native zfs encryption:

https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/

checking zfs version:

zfs-fuse being super outdated:

Importing missing zpool:

https://serverfault.com/questions/585228/zfs-import-unable-to-find-any-pools

More about ashift:

rsync stats:

https://unix.stackexchange.com/questions/65077/is-it-possible-to-see-cp-speed-and-percent-copied

09 December 2023

Updating Crucial MX500 Firmware in Linux

On my Crucial MX500, I was noticing a high level of write amplification, which is when you tell it to write 1GB of data, but it actually uses 10GB of writes to the nand flash. To try to fix this, I decided to see if a firmware update would help with this.

Identifying the Problem

Substitute /dev/sdX with you drive

Get the smart attributes

sudo smartctl -A /dev/sdX

write down the values for 247 and 248, I will refer to thus as A
wait a few days and repeat steps 1 and 2, I will refer to this as B
Now lets calculate

247C =247B - 247A
248C = 248B - 248A
(247C + 248C) / 247C

I was seeing values ranging from 10-100, when I believe the typical range should be 1-2.5

Performing the Update

Caution: Before doing any of this be sure that you have up to date backups!

Substitute /dev/sdX with your drive

Use smartctl to check what firmware version you currently have installed so you can download the correct version

sudo smartctl -i /dev/sdX
Example line: Firmware Version: M3CR020

Download the correct firmware for your device:

https://www.crucial.com/support/ssd-support/mx500-support

Mount the iso

sudo mkdir /mnt/iso
sudo mount -o loop,ro MX500_M3CR023_update.iso /mnt/iso

Create a directory to extract the files to

mkdir mx500
cd mx500

Do the extraction

gzip -dc /mnt/iso/boot/corepure64.gz | cpio -idm

List the drives

sudo ./sbin/msecli -L

Perform the Update

sudo ./sbin/msecli -U -v -i ./opt/firmware/ -n /dev/sdX

Conclusion

It seemed to help but has not completely resolved the problem

Appendix

Sources:

https://medium.com/@vdboor/upgrading-crucial-firmware-on-linux-76056254539

Others experiencing a similar write amplification issue:

Guide for calculating Write Amplification:

(PDF) https://www.micron.com/-/media/client/global/documents/products/technical-note/solid-state-storage/tnfd22_client_ssd_smart_attributes.pdf

08 December 2023

Migrating from Unifi USG-3P to UDM Pro

Decision

As my USG-3P was getting on in age and no longer getting updates, I have been shopping around for a replacement. Below were the main competitors for me.

UXG-Lite:

Price: $129 (plus tax and shipping)
WiFi: None
Unifi controller: No
CPU: Two A53 cores at 1 GHz
Memory: 1 GB of DDR3L
Storage: N/A
IPS/IDS max throughput: claimed 1Gbps (subject to third party verification)
Unifi Protect: No support

UDR:

Price: $199 (plus tax and shipping)
WiFi: 6 (however, this was to be deployed in my basement so not much help)
Unifi controller: Yes
CPU: Two A53 cores at 1.35 GHz
Memory: 2 GB of ?DDR3L?
Storage: 128 GB SSD
IPS/IDS max throughput: ~700Mbps
Unifi Protect: would need a SD card

UDM Pro:

Price: $379 --Black Friday $279-- (plus tax and shipping)
WiFi: No
Unifi controller: Yes
CPU: Four A57 cores at 1.7 GHz
Memory: 4 GB of DDR4
Storage: 16 GB eMMC
IPS/IDS max throughput: ~3.5Gbps
Unifi Protect: Hard drive slot
Bonus: LAN + WAN SFP+ ports for 10Gbps networking

To me the extra money was worth it to get the UDM Pro for the Black Friday price of $279, but I would probably not for $379. It just seems like a much more capable product that provides the option to expand later (>1Gbps networking, Unifi Protect).

Initial Impressions

Positives:

Shipping box contained lots of air pillows
High quality foam protecting the device in device packaging
Build quality is superb
Slide out foam screw holder was a nice organizational touch

Negatives:

Single use plastics used to wrap:

UDM Pro itself
Rack mount ears
Instructions, really????

Installation

Background:

This guide was written using Unifi Network 8.0.7
USG-3P network address is 192.168.1.1
Unifi controller is hosted at https://192.168.1.2:8443

Requires:

Internet connection
Laptop with an ethernet port or a PC that can be hardwired
2x ethernet cords (Only 1 is needed if you have an already hardwired PC)

Here are the steps that I used:

Create a backup from your current Unifi controller on a laptop

Navigate to https://192.168.1.2:8443
Settings -> System -> Backups
Click on `Download`
Select number of days (I chose 7)
Click `Download`

Connect a LAN port on your current network to the WAN port on UDM Pro

This is to provide it with internet access

Power on the UDM Pro
Allow it to update (this took several minutes)
Connect a laptop to the LAN port on the UDM Pro

May be helpful to disable WiFi on laptop

Setup Wizard on UDM Pro

Navigate to https://unifi/ (for me https://192.168.0.1 also would have worked)
Login with your Unifi account
Do NOT restore from Backup, skip this step
Finish the setup Wizard

Update the Network Application

You want it to be >= Unifi controller Network version

Restore the backup

Network -> Settings -> System -> Backups
Click on `Restore`
Select the backup you created above
Click on `Restore`
UDM Pro will restart

UDM Pro web UI will become unresponsive

At this point I used the touch screen to reboot the UDM Pro
However, this could be unnecessary and may possibly be resolved by forcing the laptop to get a new DHCP address

Ensure your restore happened correctly

Navigate to https://192.168.1.1
Check that the network settings are correct and that your access points are there (but they won't be connected)

Swap out the USG-3P for the UDM Pro
Migrate the Access Points from Unifi controller to UDM Pro

Navigate to https://192.168.1.2:8443
Settings -> General
Click on `Export Site` on the bottom
You can save the export file, but we won't be using it
Click continue on the `Export Site` dialog
Click continue on the `Migrate Site` dialog (no action needed)
Type in the IP address of the UDM Pro (192.168.1.1)
Select the Access Points to migrate
Click `Migrate Devices`
Check in another tab/window that they migrated to the UDM Pro
Click on `Remove Devices`

Done!

All in all, it took me about 1.5 hours, but this included unboxing, attaching the rack mount ears, mounting in the rack, and some research. The good news is that the network downtime was less than 5 minutes!

Overall Impressions

Positives:

The migration went smoothly, once I pieced together what needed done
Network performance is great
IPS has already started blocking network scans

Negatives:

Will not fit in a 12" rack that is flush mounted

AC plug is very far to one side and almost didn't have enough clearance for the 2x4 supporting my network rack, wish it was more centered
Possibly could be resolved by a 90 degree power connector

There doesn't seem to be a way to manage the screen

It will go into a screensaver mode during the day and turn off at night
However, I cannot find settings to change the times for this behavior
Update 2024-01-04:

To change this you have to set the local account as "Super Admin" on https://unifi.ui.com -> UDM Pro -> OS Settings (At the top looks like a UDM Pro with a gear icon on the bottom right) -> Admins & Users
Then you can update it https://192.168.1.1/console-settings
I set night mode to start at 10:01 PM and end at 10:00 PM thus keeping the screen off for most of the day

Other thoughts:

Lack of detailed official documentation on migration process
Topology is wrong because I have a non-Unifi switch and there isn't a way to manually fix

Research that I found:

05 October 2023

Upgrade PS4 Slim HDD to SSD

As I had finally filled my PS4's ~500 GB hard drive, it was time for an upgrade. I decided to go with a SSD in the hope that load times would improve for some of my games.

Before you start

Required parts and tools to upgrade a hard drive in a PS4 slim:

New SSD/HDD

I used a 2TB Crucial MX500 2.5" SSD that I picked up for around $100

USB hard drive big enough to backup all your applications and save files

Should be FAT or exFAT formatted
I used a 1 TB 5400 RPM hard drive

USB Flash drive at least 1 GB of size for the PS4 firmware
Screwdriver
Micro USB cable and PS4 controller (must be wired for installing system software)
Your PSN sign in information

Prep work

First, I had to format the 1 TB USB hard drive. I formatted it in Linux by following these steps:

/dev/sdd for me
remove all existing partitions using parted
add a single partition

sudo parted -a optimal /dev/sdd mkpart primary 0% 100%

format it to exFAT

sudo mkfs.exfat /dev/sdd1

Safely remove hardware:

sudo eject /dev/sdd
sudo udisksctl power-off -b /dev/sdd

Second, I prepared the USB flash drive with firmware using these steps:

Download the PS4 console reinstallation file

https://www.playstation.com/en-us/support/hardware/ps4/system-software/

Create a PS4 folder on the top level
Create a UPDATE folder in the PS4 folder
Place the PS4UPDATE.PUP file in the PS4/UPDATE/ path

Backing up your data

Then have the PS4 create a backup:

Note: took roughly 1.5 hours for me
Sync all trophies with Playstation Network
Plug the USB hard drive into the PS4
Settings -> System -> Backup and Restore
Select Backup
Set the filename
Start the backup
Note: PS4 will restart and then begin the backup

Once the backup is complete, turn off the PS4 and unplug the USB hard drive.

Physical Replacement

To replace the hard drive I followed these steps:

Note: all pictures have bottom facing up
Slide the cover off
Remove the single screw holding drive sled in place
Pull on ribbon to remove the drive sled
Remove 4 screws holding the HDD in the sled
Fasten SSD in place with the 4 screws
Slide drive sled back into place
Secure drive sled with the single screw
Slide the cover back on

Software installation and restore

Install the system software:

Plug the USB flash drive into the PS4
Turn the PS4 on in safe mode by holding the power button until you hear a second set of beeps
Plug in the controller using the USB cable
Initialize PS4 (Reinstall System Software)

Then have the PS4 restore the backup:

Note: took roughly 1.5 hours for me
Plug the USB hard drive into the PS4
Turn on the PS4
Settings -> System -> Backup and Restore
Select Restore PS4
Pick the backup file
Confirm
Note: PS4 will restart and then begin restoring the backup

After completing, restart the PS4 and remove the USB hard drive.

You will need to sign into the Playstation Network again as the process will have logged you out.

Appendix

PS4 HDD upgrade sources:

Formatting hard drive sources:

07 April 2023

Hiding mount icons for ZFS disk

As I was tired of seeing mount icons for my zfs drives on my desktop, it was time to add some more udev rules. This time i chose to set the rule by ID_SERIAL instead of KERNEL to be more confident about which drive will be ignored in case of moving them around on SATA headers.

Determine the ID_SERIAL for each of the drives:

udevadm info /dev/sdb | grep ID_SERIAL=
udevadm info /dev/sdc | grep ID_SERIAL=

Add to /etc/udev/99-hide-disks.rules (a single line for each rule):

ENV{ID_SERIAL}=="WDC_WD60EFZX-68B3FN0_WD-XXXXXXYK", ENV{UDISKS_IGNORE}="1"
ENV{ID_SERIAL}=="WDC_WD60EFZX-68B3FN0_WD-XXXXXX7K", ENV{UDISKS_IGNORE}="1"

Then have udev read from the files:

sudo udevadm control --reload

Sources:

16 January 2023

First AMD build - Ryzen 7700X

As my Intel Q6600 overclocked from 2.4GHz to 2.93 GHz was showing its age, it was time to replace this ~13 year old system that had been through many graphic card upgrades. It has also lasted thru 3 fan failures/replacements (cpu fan, back fan, and side panel fan) and the 200mm fan at the top of the Antec 900 was causing weird noises so I had it unplugged.

I have been looking at replacing it for some time (since Intel 9th gen), but had never pulled the trigger.

I purchased the Be Quiet 500FX case during their anniversary sale, as it met most of my criteria:

Good quality interior fans with at least 2 for intake and 1 for exhaust
USB C on the front panel
Glass side panel
Bottom mounted PSU with shroud for hiding/managing cables
Affordable price ($119.99, cheaper than the 500DX at the time)
The included RGB fans were a bonus
The one miss was no 5.25" bays for an optical drive, but I can use an external one

Microcenter was running 4 promotions that were tempting:

Intel 11700K + Asus Z590-Plus TUF WiFi for $299.99 (recently dropped to $249.99)
Intel 12700K + Asus Z690-Plus TUF WiFi DDR4 for $349.99
Ryzen 5600 + Gigabyte X570S AORUS Pro for $329.98
Ryzen 5700X + Gigabyte X570S AORUS Pro for $399.98
Ryzen 7700X + Asus X670E-Plus TUF WiFi + 32GB G.Skill DDR5 6000 CL36 for $679.99

The first 3 options would also require a G.Skill 32GB 3600 CL18 memory kit for $95.

The biggest draw for the last option is the promised support for new processors through at least 2025. As I am planning to use this platform for the next 10 years or so, a simple drop in processor upgrade is a major selling point. Also the PCIe 5.0 graphics and NVMe slots, allow for upgrades for the future.

The 11700K would be the highest power draw and also would not have PCIe 5.0. The 12700K has the high socket pressure that bends CPUs, so I would have to do the washer mod or replace the ILM as I think the bending would cause low reliability for the long term. The Ryzen 5600 would be only 6 cores and is the lowest performer. The Ryzen 5700X is getting really close to the Ryzen 7700X price once you factor in the memory price.

Other Items that I picked up to complete the build:

Scythe Fuma 2 Rev. B - $65.99 @ Microcenter

Dual tower with 2 fans
8 heat pipes

Western Digital SN750 SE 1TB - $99.99 @ Microcenter ($79.99 after bundle discount)

TLC flash
DDR4 cache

Corsair HX750 Platinum - $109.99 @ Newegg
Asus BW-16D1X-U - $124.99 @ Microcenter

No Bluray playing software, but I will use VLC

MSI Radeon RX 480 4GB transfered over from my existing build

Build notes:

The power supply cables could use an extra inch or two to make building easier (or perhaps cable extensions)
The 24 pin motherboard power cable was too stiff to route like normal through the cable bar and had to go diagonally through the top

cable extensions could solve this "problem"

I had to move one of the fans to fan header 1 on the Be Quiet fan controller to get the motherboard to detect the PWM control
I had to remove the back fan to be able to not scrape the motherboard on the standoffs while installing it
I should have plugged in the 2x 8 pin EPS cables before mounting the motherboard