Miscellaneous Minecraft Matters

I wanted to host a Minecraft server for me and my daughter to be able to play on. After some research, using a docker container surfaced as the easiest way to do this. Also I wanted to use Steam Link to be able to play, so I needed to add a shortcut to Minecraft in Steam.

Below are the steps that I used to accomplish this

Minecraft Bedrock Docker Server

• https://github.com/itzg/docker-minecraft-bedrock-server
• find your players XUID
• I did this by starting the server connecting and looking at the server output
• create a docker-compose.yml file where OPS has your players XUIDs in a comma separated list and ALLOW_LIST_USERS has the player names and XUIDs that you want to be able to login

version: '3.4'

services:

    bds:

        image: itzg/minecraft-bedrock-server

        environment:

            EULA: "TRUE"

            GAMEMODE: creative

            DIFFICULTY: peaceful

            SERVER_NAME: "Our World"

            OPS: "1234,5678"

            ALLOW_CHEATS: "true"

            ALLOW_LIST: "true"
            ALLOW_LIST_USERS: "player1:1234,player with spaces:5678"

        ports:

            - "19132:19132/udp"

        volumes:

            - /storage/containers/minecraft/world1:/data

        stdin_open: true

        tty: true

• start the container
• docker-compose up
• A permissions.json file will be created giving the specified players ops powers
• Note: even though your server is local the Playstation/Xbox/Switch version will not be able to connect without a PS Plus/Xbox Live/Nintendo Online subscription
• https://bugs.mojang.com/browse/BDS-1634

Adding a shortcut to Minecraft in Steam

• Find out where Minecraft was stored
• Paste the following into an explorer window:
• %LocalAppData%\Packages\
• Find the folder like:
• Microsoft.MinecraftUWP_<seemingly_random_letters_and_numbers>
• The seemingly random letters and numbers are the app id, we will need them for later
• In Steam go to your Library and click "ADD A GAME" and then "Add a Non-Steam Game..."
• Navigate to C:\Windows and select explorer.exe
• You will see a new entry in your library called explorer
• Right click on it -> Properties
• Choose an appropriate icon
• Rename it
• Click "SET LAUNCH OPTIONS"
• type/paste in the following:
• shell:appsFolder\<your-app-id>!App
• Click "OK"
• Click "CLOSE"
• You should now be able to launch Minecraft from Steam

Appendix

Sources:

• https://linustechtips.com/topic/547779-add-universal-apps-and-purchased-windows-store-games-to-steam-library-and-how-to-pin-steam-games-to-windows-taskbar-and-start-menu/

Changing a zpool from ashift=9 to ashift=12

I wanted the additional write speed on my nas drives that come from aligning the ashift value with physical sector size of my hard drives (ashift=9 is 512 bytes and ashift=12 is 4KB). Unfortunately, you cannot change ashift on an existing zpool, so you will have to backup the data, destroy the pool, recreate it, and then restore the data.

Prereq

• pv (to monitor the process/speed)
• sudo apt-get install pv
• encrypted zfs data with the "wrong" ashift value that you want to migrate

Process to move

1. Stop any process that writes to your storage that you want to move
2. Setup temporary storage location
• sudo zpool create external-storage mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXX /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXX
• sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase external-storage/encrypted
3. Create a snapshot
• sudo zfs snapshot storage/encrypted@migrate-20231229
• sudo zfs list -t snapshot
4. Copy snapshot over
• sudo bash -c 'zfs send storage/encrypted@migrate-20231229 | pv | zfs recv external-storage/encrypted/backup'
5. Ensure that all files have been backed up
6. Unmount the datasets
• sudo zfs unmount storage/encrypted
7. Destroy the old zpool
• sudo zpool destroy storage
8. Create the new zpool
• sudo zpool create storage mirror /dev/disk/by-id/ata-WDC_WD60EFZX-68B3FN0_WD-XXXXXXX ata-WDC_WD60EFZX-68B3FN0_WD-XXXXXXX
9. Ensure is setup with the correct ashift value
• sudo zdb -C storage | grep ashift
10. Create a temporary file to contain your passphrase because since zfs recv is using stdin to pull in the data it cannot prompt for it
• echo "super-secret" > /home/example/passphrase.txt
11. Copy snapshot back
• sudo bash -c 'zfs send external-storage/encrypted/backup@migrate-20231229 | pv | zfs recv -o encryption=aes-256-gcm -o keylocation=file:///home/example/passphrase.txt -o keyformat=passphrase storage/encrypted'
12. Change from a file to prompt for password
• sudo zfs change-key -o keylocation=prompt storage/encrypted
13. Remove the temp passphrase
• rm /home/example/passphrase.txt
14. Check that all your files are back in place
15. Now if you want you can destroy the backup or export it and keep the backup
• sudo zpool destroy external-storage
• OR
• sudo zpool export external-storage

Appendix

If you see an error like:

• cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
• That means that you cannot copy to the encrypted dataset. What I did to get around this was to instead copy to a child of the encrypted dataset.

Sources:

• https://gist.github.com/kapythone/dbdbf7ad1961c606e62e215e5d30de98
• https://mtlynch.io/zfs-encrypted-backups/
• https://unix.stackexchange.com/questions/532619/encryption-of-existing-dataset-in-zfs-zol-0-8/575439#575439
• https://charsiurice.wordpress.com/2016/05/30/checking-ashift-on-existing-pools/

Migrating from eCryptFS to native zfs encryption

I wanted to move from eCryptFS on top of a zfs dataset to a more standard and speedier encryption approach which is native zfs encryption. Here is the process that I went through.

Process

1. Ensure your backups are up to date!
2. Upgrade the zpool
• ensure you are on a recent version of zfs and not zfs-fuse (see previous post)
• sudo zpool upgrade storage
3. Create the destination dataset
• sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase storage/new-encrypt
4. Set/change the mount point (optional)
• sudo zfs set mountpoint=/storage/new-encrypt storage/new-encrypt
5. Move the files over
• sudo rsync -avh --progress --remove-source-files /storage/encrypted/* /storage/new-encrypt/
• -z / --compress is not needed and would slow down a local transfer
6. Remove the left over directories
• sudo find /storage/encrypted/ -type d -empty -delete
7. Verify no files are left:
• ls -al /storage/encrypted
• if any files exist then repeat the rsync
8. Unmount the encryptfs
• sudo umount /storage/encrypted
9. Remove/comment the entry from /etc/fstab
• sudo vi /etc/fstab
10. Unmount the zfs dataset
• sudo zfs unmount storage/.encrypted
11. Test destroying the zfs dataset
• sudo zfs destroy -n storage/.encrypted
12. Destroy the zfs dataset
• sudo zfs destroy storage/.encrypted
13. Change the name of new-encrypt
• sudo zfs rename storage/new-encrypt storage/encrypted
14. Update mountpoint (if required)
• sudo zfs set mountpoint=/storage/encrypted storage/encrypted

If everything works then the new zfs native encrypted dataset slots right into where the old one was and all your samba shares should be fine.

Appendix

Sources:

• https://rucore.net/en/luks-ecryptfs-or-zfs-encryption-choosing-a-way-to-protect-your-data-in-linux/

Adventures with ZFS

As part of plan to improve my backup strategy, I was testing out different zfs configurations to see what might be the best option.

The Problem

I was having trouble testing out using native zfs encryption

• sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt
• invalid property 'encryption'

Checking zfs version

So first I checked the zfs version:

• sudo modinfo zfs | grep version
• version:        0.8.3-1ubuntu12.15
• zpool version
• returned an error, which I found weird
• zpool upgrade -v
• returned a maximum version of 23
• expected a maximum version of 28
• Did some googling and found out that I may have the older zfs-fuse installed
• dpkg -s zfs-fuse 
• confirmed my suspicion and told me that I had 0.7.0 installed

Replace zfs-fuse with zfsutils-linux

• * Unmount all zfs datasets*
• I didn't do this step, but you definitely should to prevent data errors
• sudo zfs unmount <zpool>/<dataset>
• sudo apt remove zfs-fuse
• sudo apt install zfsutils-linux
• Here I rebooted the system
• sudo zpool status
• no zpools available
• sudo zpool import -a
• listed my zpool with its last use
• sudo zpool import -f <zpool>
• sudo zpool status
• now correctly showed my zpool

Test setup

• I am using 2 disks that have a raw speed of 150 MB/s (benchmarked using dd on a single disk)
• Creating the test pool
• sudo zpool create test-pool mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>
• or
• sudo zpool create test-pool -o ashift=9 mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number> /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-<serial_number>
• Creating test dataset
• sudo zfs create test-pool/test
• sudo zfs create -o encryption=aes-256-gcm -o keylocation=prompt -o keyformat=passphrase test-pool/test-encrypt
• Disable caching
• sudo zfs set primarycache=none test-pool/test
• sudo zfs set secondarycache=none test-pool/test
• 10 back-to-back copies of a 5GB file and waiting for the numbers to stabilize
• sudo rsync --progress Downloads/Win11_22H2_English_x64v1.iso /test-pool/test/
• Deleting the zpool when done testing
• sudo zpool destroy test-pool

Results of the upgrade and test

The good news is that I am seeing a 2-3X speed up on my existing zpool setup with ashift=9 from doing the upgrade. It also appears that native encryption has a minimal impact (as long as your processor has AES-NI). Here are the numbers that I was seeing

• zfs-fuse 0.7.0:
• ashift=9, no encryption: 35-45 MB/s
• ashift=12, no encryption:  70-72 MB/s
• zfsutils-linux 0.8.3:
• ashift=9, no encryption: 108-117 MB/s
• ashift=9, native encryption: 105-117 MB/s
• ashift=12, no encryption: 134-141 MB/s
• ashift=12, native encryption: 133-139 MB/s

Appendix

Article on setting up native zfs encryption:

• https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/

checking zfs version:

• https://www.reddit.com/r/zfs/comments/8bxxhu/how_do_i_check_which_version_of_zfs_i_have/
• https://askubuntu.com/questions/423355/how-do-i-check-if-a-package-is-installed-on-my-server

zfs-fuse being super outdated:

• https://github.com/openzfs/zfs/issues/10329
• https://github.com/pscedu/zfs-fuse

Importing missing zpool:

• https://serverfault.com/questions/585228/zfs-import-unable-to-find-any-pools

More about ashift:

• https://jro.io/truenas/openzfs/#tuning
• https://louwrentius.com/zfs-performance-and-capacity-impact-of-ashift9-on-4k-sector-drives.html

rsync stats:

• https://unix.stackexchange.com/questions/65077/is-it-possible-to-see-cp-speed-and-percent-copied

Updating Crucial MX500 Firmware in Linux

On my Crucial MX500, I was noticing a high level of write amplification, which is when you tell it to write 1GB of data, but it actually uses 10GB of writes to the nand flash.  To try to fix this, I decided to see if a firmware update would help with this. 

Identifying the Problem

Substitute /dev/sdX with you drive

1. Get the smart attributes
• sudo smartctl -A /dev/sdX
2. write down the values for 247 and 248, I will refer to thus as A
3. wait a few days and repeat steps 1 and 2, I will refer to this as B
4. Now lets calculate
1. 247C =247B - 247A
2. 248C = 248B - 248A
3. (247C + 248C) / 247C
5. I was seeing values ranging from 10-100, when I believe the typical range should be 1-2.5

Performing the Update

Caution: Before doing any of this be sure that you have up to date backups!

Substitute /dev/sdX with your drive

• Use smartctl to check what firmware version you currently have installed so you can download the correct version
• sudo smartctl -i /dev/sdX
• Example line: Firmware Version: M3CR020
• Download the correct firmware for your device:
• https://www.crucial.com/support/ssd-support/mx500-support
• Mount the iso
• sudo mkdir /mnt/iso
• sudo mount -o loop,ro MX500_M3CR023_update.iso /mnt/iso
• Create a directory to extract the files to
• mkdir mx500
• cd mx500
• Do the extraction
• gzip -dc /mnt/iso/boot/corepure64.gz | cpio -idm
• List the drives
• sudo ./sbin/msecli -L
• Perform the Update
• sudo ./sbin/msecli -U -v -i ./opt/firmware/ -n /dev/sdX

Conclusion

It seemed to help but has not completely resolved the problem

Appendix

Sources:

• https://medium.com/@vdboor/upgrading-crucial-firmware-on-linux-76056254539

Others experiencing a similar write amplification issue:

• https://www.reddit.com/r/buildapc/comments/efyo32/write_amplification_problem_with_mx500_500_gb/
• https://www.techpowerup.com/forums/threads/mx500-rapid-erase-cycles-seems-to-be-poor-wear-levelling.287023/page-4

Guide for calculating Write Amplification:

• (PDF) https://www.micron.com/-/media/client/global/documents/products/technical-note/solid-state-storage/tnfd22_client_ssd_smart_attributes.pdf

Migrating from Unifi USG-3P to UDM Pro

Decision

As my USG-3P was getting on in age and no longer getting updates, I have been shopping around for a replacement. Below were the main competitors for me.

UXG-Lite:

• Price: $129 (plus tax and shipping)
• WiFi: None
• Unifi controller: No
• CPU: Two A53 cores at 1 GHz
• Memory: 1 GB of DDR3L
• Storage: N/A 
• IPS/IDS max throughput: claimed 1Gbps (subject to third party verification)
• Unifi Protect: No support

UDR:

• Price: $199 (plus tax and shipping)
• WiFi: 6 (however, this was to be deployed in my basement so not much help)
• Unifi controller: Yes
• CPU: Two A53 cores at 1.35 GHz
• Memory: 2 GB of ?DDR3L?
• Storage: 128 GB SSD
• IPS/IDS max throughput: ~700Mbps
• Unifi Protect: would need a SD card

UDM Pro:

• Price: $379 --Black Friday $279-- (plus tax and shipping)
• WiFi: No
• Unifi controller: Yes
• CPU: Four A57 cores at 1.7 GHz
• Memory: 4 GB of DDR4
• Storage: 16 GB eMMC
• IPS/IDS max throughput: ~3.5Gbps
• Unifi Protect: Hard drive slot
• Bonus: LAN + WAN SFP+ ports for 10Gbps networking

To me the extra money was worth it to get the UDM Pro for the Black Friday price of $279, but I would probably not for $379. It just seems like a much more capable product that provides the option to expand later (>1Gbps networking, Unifi Protect).

Initial Impressions

Positives:

• Shipping box contained lots of air pillows
• High quality foam protecting the device in device packaging
• Build quality is superb
• Slide out foam screw holder was a nice organizational touch

Negatives:

• Single use plastics used to wrap:
• UDM Pro itself
• Rack mount ears
• Instructions, really????

Installation

Background:

• This guide was written using Unifi Network 8.0.7
• USG-3P network address is 192.168.1.1
• Unifi controller is hosted at https://192.168.1.2:8443

Requires:

• Internet connection
• Laptop with an ethernet port or a PC that can be hardwired
• 2x ethernet cords (Only 1 is needed if you have an already hardwired PC)

Here are the steps that I used:

1. Create a backup from your current Unifi controller on a laptop
1. Navigate to https://192.168.1.2:8443
2. Settings -> System -> Backups
3. Click on `Download`
4. Select number of days (I chose 7)
5. Click `Download`
2. Connect a LAN port on your current network to the WAN port on UDM Pro
• This is to provide it with internet access
3. Power on the UDM Pro
4. Allow it to update (this took several minutes)
5. Connect a laptop to the LAN port on the UDM Pro
• May be helpful to disable WiFi on laptop
6. Setup Wizard on UDM Pro
1. Navigate to https://unifi/ (for me https://192.168.0.1 also would have worked)
2. Login with your Unifi account
3. Do NOT restore from Backup, skip this step
4. Finish the setup Wizard
7. Update the Network Application
• You want it to be >= Unifi controller Network version
8. Restore the backup
1. Network -> Settings -> System -> Backups
2. Click on `Restore`
3. Select the backup you created above
4. Click on `Restore`
5. UDM Pro will restart
9. UDM Pro web UI will become unresponsive
• At this point I used the touch screen to reboot the UDM Pro
• However, this could be unnecessary and may possibly be resolved by forcing the laptop to get a new DHCP address
10. Ensure your restore happened correctly
1. Navigate to https://192.168.1.1
2. Check that the network settings are correct and that your access points are there (but they won't be connected)
11. Swap out the USG-3P for the UDM Pro
12. Migrate the Access Points from Unifi controller to UDM Pro
1. Navigate to https://192.168.1.2:8443
2. Settings -> General
3. Click on `Export Site` on the bottom
4. You can save the export file, but we won't be using it
5. Click continue on the `Export Site` dialog
6. Click continue on the `Migrate Site` dialog (no action needed)
7. Type in the IP address of the UDM Pro (192.168.1.1)
8. Select the Access Points to migrate
9. Click `Migrate Devices`
10. Check in another tab/window that they migrated to the UDM Pro
11. Click on `Remove Devices`
13. Done!

All in all, it took me about 1.5 hours, but this included unboxing, attaching the rack mount ears, mounting in the rack, and some research. The good news is that the network downtime was less than 5 minutes!

Overall Impressions

Positives:

• The migration went smoothly, once I pieced together what needed done
• Network performance is great
• IPS has already started blocking network scans

Negatives:

• Will not fit in a 12" rack that is flush mounted
• AC plug is very far to one side and almost didn't have enough clearance for the 2x4 supporting my network rack, wish it was more centered
• Possibly could be resolved by a 90 degree power connector
• There doesn't seem to be a way to manage the screen
• It will go into a screensaver mode during the day and turn off at night
• However, I cannot find settings to change the times for this behavior
• Update 2024-01-04:
• To change this you have to set the local account as "Super Admin" on https://unifi.ui.com -> UDM Pro -> OS Settings (At the top looks like a UDM Pro with a gear icon on the bottom right) -> Admins & Users
• Then you can update it https://192.168.1.1/console-settings
• I set night mode to start at 10:01 PM and end at 10:00 PM thus keeping the screen off for most of the day

Other thoughts:

• Lack of detailed official documentation on migration process
• Topology is wrong because I have a non-Unifi switch and there isn't a way to manually fix

Research that I found:

• https://www.reddit.com/r/UNIFI/comments/yfyrib/good_current_guide_to_migrate_to_a_new_controller/
• https://fydelia.zendesk.com/hc/en-us/articles/360011466220-Migrating-UniFi-APs-to-another-controller
• https://help.ui.com/hc/en-us/articles/360008976393-UniFi-How-to-Migrate-from-Cloud-Key-to-Cloud-Key-or-UDM
• https://reimling.eu/2022/01/migrate-ubiquiti-unifi-security-gateway-usg-to-unifi-dream-machine-pro-udm-pro/
• https://community.ui.com/questions/Unable-to-uninstall-install-applications-on-UDM-Pro/54f23f0e-9500-44fc-8516-c130986ff24a

Upgrade PS4 Slim HDD to SSD

As I had finally filled my PS4's ~500 GB hard drive, it was time for an upgrade. I decided to go with a SSD in the hope that load times would improve for some of my games.

Before you start

Required parts and tools to upgrade a hard drive in a PS4 slim:

• New SSD/HDD
• I used a 2TB Crucial MX500 2.5" SSD that I picked up for around $100
• USB hard drive big enough to backup all your applications and save files
• Should be FAT or exFAT formatted
• I used a 1 TB 5400 RPM hard drive
• USB Flash drive at least 1 GB of size for the PS4 firmware
• Screwdriver
• Micro USB cable and PS4 controller (must be wired for installing system software)
• Your PSN sign in information

Prep work

First, I had to format the 1 TB USB hard drive. I formatted it in Linux by following these steps:

• /dev/sdd for me
• remove all existing partitions using parted
• add a single partition
• sudo parted -a optimal /dev/sdd mkpart primary 0% 100%
• format it to exFAT
• sudo mkfs.exfat /dev/sdd1
• Safely remove hardware:
• sudo eject /dev/sdd
• sudo udisksctl power-off -b /dev/sdd

Second, I prepared the USB flash drive with firmware using these steps:

• Download the PS4 console reinstallation file
• https://www.playstation.com/en-us/support/hardware/ps4/system-software/
• Create a PS4 folder on the top level
• Create a UPDATE folder in the PS4 folder
• Place the PS4UPDATE.PUP file in the PS4/UPDATE/ path

Backing up your data

Then have the PS4 create a backup:

• Note: took roughly 1.5 hours for me
• Sync all trophies with Playstation Network
• Plug the USB hard drive into the PS4
• Settings -> System -> Backup and Restore
• Select Backup
• Set the filename
• Start the backup
• Note: PS4 will restart and then begin the backup

Once the backup is complete, turn off the PS4 and unplug the USB hard drive.

Physical Replacement

To replace the hard drive I followed these steps:

• Note: all pictures have bottom facing up
• Slide the cover off
  
• Remove the single screw holding drive sled in place
  
• Pull on ribbon to remove the drive sled
  
• Remove 4 screws holding the HDD in the sled
• Fasten SSD in place with the 4 screws
• Slide drive sled back into place
• Secure drive sled with the single screw
• Slide the cover back on

Software installation and restore

Install the system software:

• Plug the USB flash drive into the PS4
• Turn the PS4 on in safe mode by holding the power button until you hear a second set of beeps
• Plug in the controller using the USB cable
• Initialize PS4 (Reinstall System Software)

Then have the PS4 restore the backup:

• Note: took roughly 1.5 hours for me
• Plug the USB hard drive into the PS4
• Turn on the PS4
• Settings -> System -> Backup and Restore
• Select Restore PS4
• Pick the backup file
• Confirm
• Note: PS4 will restart and then begin restoring the backup

After completing, restart the PS4 and remove the USB hard drive.

You will need to sign into the Playstation Network again as the process will have logged you out.

Appendix

PS4 HDD upgrade sources:

• https://www.playstation.com/en-us/support/hardware/ps4-upgrade-hdd/
• https://www.playstation.com/en-us/support/hardware/ps4/system-software/
• https://www.playstation.com/en-us/support/hardware/ps4-back-up-and-restore-with-external-storage/

Formatting hard drive sources:

• https://unix.stackexchange.com/questions/38164/create-partition-aligned-using-parted
• https://unix.stackexchange.com/questions/551030/why-does-parted-need-a-filesystem-type-when-creating-a-partition-and-how-does-i
• https://unix.stackexchange.com/questions/35508/eject-usb-drives-eject-command