Tech Tips

03 November 2024

Ubuntu VM add storage

Since my Ubuntu VM was running out of space, I needed to expand it.

Steps I used

Expand the storage in Proxmox

Proxmox web GUI -> Select the VM -> Hardware
Select the Hard Disk
Disk Action -> Resize
Type the number of GB to add -> Resize disk

Have the VM use the additional storage

Proxmox web GUI -> Select the VM -> Console
login
sudo cfdisk /dev/sda

sort (This should hopefully put your FreeSpace right after your Linux filesystem
resize
Write

Reboot
sudo resize2fs /dev/sda4

Appendix

Sources

https://askubuntu.com/questions/116351/increase-partition-size-on-which-ubuntu-is-installed

Repair La-Z-Boy Office Chair Lumbar support

Similar to https://www.reddit.com/r/fixit/comments/1btyyg3/lazboy_ergonamic_mesh_chair_lumbar_pad_came_off/ my lumbar support fell off on my La-Z-Boy Ergonomic Mesh Swivel Task Chair model 60021 (Joel). The lumbar support was always loose and had to constantly be readjusted to where I wanted it.

The problem

The single screw that holds the padding to the lumbar support arm backed all the way out which caused it to fall off.
There is no apparent way to easily reattach it

The solution

remove the arm the padding attaches to by removing the 2 screws with an allen key
separate the padding from the plastic backing by carefully pulling them apart until the 4 posts pulled out of the padding (one in each corner)
attach the plastic backing to the arm using the plastic adjuster, locking washer, and screw

I screwed it in really tight, but if you have some thread lock you may want to use it

snap the padding back onto the plastic backing by pressing it into place
reattach the arm to the chair

The result

The lumbar support is reattached and now much better at staying in place (better than new)

Update: Since I did not use thread lock the lumbar support has loosened some

14 September 2024

Adding a USB card to Dell r730xd

As I was tired of being limited to just 2 USB ports, I picked up the Inateck RedComets U21 20Gbps on Prime Day for $31.99 plus tax.

Specs

2 lanes of PCIe 2.0/3.0/4.0 (3.0 or 4.0 required for full speed)
Max combined throughput 16 Gbps
10 Gbps dedicated port

USB-A

10 Gbps split between

2 x USB-A
2 x USB-C

Add USB Card

Shutdown containers and VMs
Turn off the server
Install the USB card

I installed it above the 2.5" drive bays in the back in Riser 3 (Slot 6)

Turn on the server
Fix fan speed

apt-get install ipmitool
ipmitool -I lanplus -H 192.168.1.X -U root -P <password> raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00
response:

16 05 00 00 00

Shutdown the VM you want to add it to
Add the USB Card to a VM

Select your VM -> Hardware -> Add -> PCI Device
Select Raw Device -> Device: ASM2142/ASM3142 USB 3.1 Host Controller
Check All Functions
Add

Start the VM
Check that it is working

lsusb -t

Test a USB hard drive

plug in the USB hard drive
determine what drive letter it is:

ls -l /dev/disk/by-id

Mount it

sudo mount -m /dev/sdb1 /mnt/backup-plus

Error: unknown filesystem type 'exfat'

sudo apt-get install exfat-fuse
sudo mount /dev/sdb1 /mnt/backup-plus -t exfat-fuse

Appendix

Sources

Setting up a Jellyfin media server

I wanted to install a media server to help manage my growing library. The top two options that I found are Plex and Jellyfin. I have used Plex in the past, but disliked when they started requiring an account so I went with Jellyfin.

Create the container

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/jellyfin.sh)"

Set Static IP and hostname

Proxmox -> jellyfin -> Network -> net0 -> Edit

set static ip

Restart the container
Proxmox -> jellyfin -> Console

ping 192.168.1.1

this ensures that the container will show up on Unifi's list of clients

Unifi -> Clients -> Select the jellyfin -> Settings

give it a memorable name
check "Fixed IP Address"

it should auto-populate with current IP

check "Local DNS Record"

jellyfin.home.arpa

Setup Jellyfin

Self signed SSL certificate:

https://jellyfin.org/docs/general/networking/
jellyfin -> Console

mkdir -p /home/jellyfin
cd /home/jellyfin
openssl req -x509 -days 3650 -newkey rsa:4096 -keyout ./privkey.pem -out cert.pem -nodes -subj '/CN=jellyfin.home.arpa'

-subj '/' can work if you can override ssl verification on the client, but Kodi doesn't have that option
-subj '/CN=localhost' was causing jellyfin to crash here are some keywords from the crash:

Error occurred during a cryptographic operation
MapOpenSsl30Code

openssl pkcs12 -export -out jellyfin.pfx -inkey privkey.pem -in cert.pem -passout pass:
chown -R jellyfin /home/jellyfin
chgrp -R jellyfin /home/jellyfin

http://jellyfin.home.arpa:8096

create user/pass
hamburger menu -> Dashboard -> Networking

Check "Enable HTTPS"
Check "Require HTTPS"
Custom SSL certificate path -> /home/jellyfin/jellyfin.pfx
Save

Restart the jellyfin container
Test the self-signed certificate

curl --cacert cert.pem https://jellyfin.home.arpa:8920

https://jellyfin.home.arpa:8920

hamburger menu -> Dashboard -> Users

Add User
Name
Password
Check "Enable access to all libraries"

hamburger menu -> Dashboard -> Plugin -> Catalog -> Kodi Sync Queue

Install

Restart jellyfin

Add to Kodi on Google TV

Have Kodi trust the self signed certificate
Enable adb (Android Debug Bridge)

Determine IP of the Google TV

Settings -> System -> About -> Status

Enable Developer mode

Settings -> System -> About -> Build Number -> Click 10 times or until developer mode is unlocked

Enable USB debugging (also enables remote debugging)

install adb onto a computer to be able to modify the Google TV remotely

sudo apt install android-tools-adb

Used adb to add new certs file to existing

adb pull /sdcard/Android/data/org.xbmc.kodi/files/.kodi/addons/script.module.certifi/lib/certifi/cacert.pem
cat cacert.pem cert.pem > updatedcacert.pem
adb push updatedcacert.pem /sdcard/Download/
echo -e '<advancedsettings version="1.0">\n<network>\n<catrustfile>special://masterprofile/updatedcacert.pem</catrustfile>\n</network>\n</advancedsettings>' >advancedsettings.xml
adb push advancedsettings.xml /sdcard/Download/

Have Kodi use the new certs

Google TV -> Settings -> Apps -> Kodi -> Permissions -> Files and Media -> Allow management of all files
Kodi -> Settings -> File Manager

Left Side

Add Source -> Browse -> External Storage -> Download -> OK
Open Download

Right Side

Profile directory

Long press the files one at a time in the Left pane and select copy

Restart Kodi

Install Jellyfin on Kodi

Kodi -> Settings -> File Manager -> Add Source

https://kodi.jellyfin.org

Kodi -> Settings -> AddOn Browser

Allow install from unknown sources when prompted
Install from zipfile -> jellyfin -> repository.jellyfin.kodi.zip
Install from Repository -> Kodi Jellyfin Add-ons -> Video Add-ons
Select Jellyfin add-on and install

Cancel adding the server
Disable "Verifiy connection" -> OK
Restart Kodi
Select Manual server add (as auto does it by IP)

https://jellyfin.home.arpa:8920
sign in using user/pass
Playback -> Addon (default)
Choose which libraries you want to sync

click "All"
click "OK"

After Syncing is complete restart Kodi
Settings -> Add-ons -> My add-ons -> Video add-ons -> Jellyfin

Configure -> Sync -> Enable Kodi Sync Queue -> OK

Appendix

Sources

samba

Jellyfin

kodi

curl

https://stackoverflow.com/questions/9879688/difference-between-cacert-and-capath-in-curl

ssl certificate

https://ruk.ca/content/setting-self-signed-certificate-expiration-date

18 July 2024

Debugging Windows 11 crashes

Once to twice a week, I have been finding my system asleep when it should have been folding. After looking at "Event Viewer" after each of these I noticed a trend of a Kernel-Power event proceeded (but not immediately) by a volmgr one.

To view the events open up "Event Viewer", expand "Windows Logs", and click on "System"

Simplified Example:

Level	Date and Time	Source	Event ID	Task Category
Critical	5/7/2024 2:13:36 AM	Kernel-Power	41	(63)
	The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.
Error	5/7/2024 2:13:36 AM	volmgr	162	None
	Dump file generation succeded.
Warning	5/7/2024 2:11:00 AM	Display	4101	None
	Display driver amduw23g stopped responding and has successfully recovered.
Warning	5/7/2024 2:06:07 AM	Display	4101	None
	Display driver amduw23g stopped responding and has successfully recovered.

Updating Drivers

As a first step to try and remedy this, I always update drivers. I updated the following:

chipset:

4.07.13.2243 -> 5.11.02.217 (from Asus)
6.02.07.2300 (from AMD after above didn't fix it)

video: a version installed in 2024 -> 24.5.1

Updating the BIOS

I found some posts online that said it could be due to the motherboard, so I decided to try updating the BIOS to see if that would help. Unfortunately, it did not.

Before updating the BIOS/UEFI make sure to write down your changes as they will be reset.

BIOS/UEFI Settings

Expo -> Enabled
Fan Curves

20C -> 20%
60C -> 40%
85C -> 70%
90C -> 100%

System

20C -> 30%
50C -> 50%
85C -> 80%
90C -> 100%

Advanced Mode (F7)

Tool -> ASUS Armoury Crate

Download & Install -> Disabled

AI Tweaker -> Precision Boost Override

Curve Optimizer

All Cores
Negative
30

Precision Boost Override -> AMD Eco Mode
AMD Eco Mode -> cTDP 65W

AI Tweaker -> SOC Offset -> negative -> 0.03 (however this caused the pc to hang on warm-boots so I use the below settings instead)
AI Tweaker -> CPU SOC Voltage -> Manual

VDD SOC Voltage Override -> 1.2

Update the BIOS

https://www.asus.com/motherboards-components/motherboards/tuf-gaming/tuf-gaming-x670e-plus-wifi/helpdesk_bios/?model2Name=TUF-GAMING-X670E-PLUS-WIFI
1618 (2023-05-25) -> 2613 (2024-04-17)

After updating the BIOS, you will need to reapply the settings.

Old video drivers

Since all the updates didn't work, I tried installing old video drivers that I knew worked from 24.5.1 to 23.10.2. Luckily this seems to have done the trick and my system is stable again with >27 days of uptime.

Slow folding

Unfortunately, it looks like my solution may be short lived as to get expected folding performance, I need to update to 24.6.1: https://foldingforum.org/viewtopic.php?t=41637&sid=d9b1c6f33f52801aaca8c31fc3fe52d1 So fingers crossed that this release is also stable.

Update 2024-07-29: I had a freeze after roughly 11 days of uptime

Update 2024-07-31: My PC crashed overnight after only 1.5 days of uptime

24.7.1

Update 2024-07-31: I updated the graphics drivers to 24.7.1. Here is the process that I used:

Downloaded the AMD driver cleanup utility (from here)
Downloaded the full updated AMD graphics driver (for 5600 XT)
Disabled Ethernet/Wi-Fi

I do this so that Windows doesn't try to "helpfully" download and install other graphic driver versions

Ran the AMD driver cleanup utility

Had it reboot me into Safe-Mode
It removed the drivers and prompted me to reboot, which I did

Installed the new drivers (had to approve the big red scary box because of no internet)
Rebooted
Enabled Ethernet/Wi-Fi

Update 2024-08-02: My computer has crashed twice in the past 2 days so I decided to try DDU

Downloaded Display Driver Uninstaller (aka DDU)

I downloaded from Guru3D

Disabled Ethernet/Wi-Fi
Rebooted into Safe Mode

https://support.microsoft.com/en-us/windows/start-your-pc-in-safe-mode-in-windows-92c27cff-db89-8644-1ce4-b3e5e56fe234
Settings -> System -> Recovery

next to "Advanced startup", select "Restart now"

System will reboot
Troubleshoot -> Advanced options -> Startup Settings -> Restart
System will reboot again
Type "4" for Safe Mode

Extracted the EXE from the ZIP file
Ran the EXE, it will extract even more files
Ran "Display Driver Uninstaller.exe"
Unchecked the final option (Disable Windows Update Driver Downloads)

Since we already disabled internet access this is unnecessary and DDU recommends reenabling afterwards anyway so this saves a step

Close the options
Device Type -> GPU
Device -> AMD
Clicked "Clean and restart"
Waited for it to do its thing and reboot
Installed the 24.7.1 drivers again
Rebooted
Enabled Ethernet/Wi-Fi

Appendix

Sources

15 July 2024

Intel SSD upgrade for r730xd

I got 2 used intel ssds to improve VM performance and to reduce power usage

Check SMART attributes

General Health

smartctl -H /dev/sdX

Wearout

100 is best aka no wear (https://www.solidigm.com/support-page/product-doc-cert/ka-00033.html)
smartctl -A /dev/sdX | grep Wearout

General

smartctl -A /dev/sdX

Lucky for me my drives came at 89% and 90% life left with ~60K hours and ~1PiB of Host writes out of the Intel spec (pdf) of 5.3PB

Firmware Update Adventure

Intel Memory and Storage Tool CLI

My first attempt was to use Intel cli to update the firmware, but it did not work.

Installation

https://www.intel.com/content/www/us/en/download/19520/intel-memory-and-storage-tool-cli-command-line-interface.html
mkdir intelmas && cd intelmas
wget https://downloadmirror.intel.com/822151/Intel_MAS_CLI_Tool_Linux_2.5.zip
unzip Intel_MAS_CLI_Tool_Linux_2.5.zip
dpkg -i intelmas_2.5.0-0_amd64.deb

Usage

https://forums.servethehome.com/index.php?threads/intelmas-quick-guide.36165/
intelmas show -intelssd

Solidigm: This drive is no longer supported.

intelmas show -a -intelssd 4

EnduranceAnalyzer : Drive is not an Intel SSD.

unfortunately, it listed the SSDs as not supported, but it did point me Solidigm

Solidigm

My second attempt was to use Solidigm cli to update the firmware, but it did not work through the raid controller and had to be attached to a different SATA controller.

Installation

https://www.solidigm.com/support-page/drivers-downloads/ka-00085.html
mkdir solidigm && cd solidigm
wget https://sdmsdfwdriver.blob.core.windows.net/files/kba-gcc/drivers-downloads/ka-00085/sst--1-13/sst-cli-linux-deb--1-13.zip
unzip sst-cli-linux-deb--1-13.zip
dpkg -i sst_1.13.288-0_amd64.deb

Usage

https://sdmsdfwdriver.blob.core.windows.net/files/kba-gcc/drivers-downloads/ka-00085/sst--1-13/solidigm-storage-tool-cli-user-guide-727329-010us.pdf
sst show -ssd
sst show -a -ssd <index> | grep Firmware

Firmware Update

sst load -ssd <index>
Status : Firmware update failed.

If your drives are behind a RAID card, you may have to remove them for updating.

Firmware Update using a USB to SATA enclosure

sst show -ssd
Unfortunately, the drive no longer appeared :-(

Firmware Update using Windows with USB to SATA enclosure

The drive appeared and would run tests
However attempting to update the firmware resulted in an error and the USB port being disabled until the PC was rebooted

Firmware Update using a JMB585 PCIe to SATA card, a SATA to eSATA bracket, and an external eSATA enclosure

Finally was able to update the firmware
Notes:

the firmware did not jump to the latest version and had to be updated incrementally

G2010140 -> G2010160 -> G2010170

the firmware update requested a reboot after installing so I did that

sst show -ssd
sst show -a -ssd <index> | grep Firmware
sst load -ssd <index>
Status : Firmware updated successfully. Please reboot the system.

Replacing the existing 1.2 TB drives

Setup the partitions

I used my rpool mirror instructions from here but made the following changes

removed the last-lba line
removed the start from all lines
removed the size from the final partition
don't run the zpool attach

Copy the rpool data

I am following the wizardry provided here
Add a mirror of the new disks to form a striped mirror

zpool add rpool mirror /dev/disk/by-id/ata-INTEL_SSDSC2BX800G4_<SERIAL1>-part3 /dev/disk/by-id/ata-INTEL_SSDSC2BX800G4_<SERIAL2>-part3

Check the mirror names and size

zpool status rpool
zpool list rpool

Remove the old mirror

zpool remove rpool mirror-0

Check for when "Evacuation of mirror" is done

zpool status rpool
remove: Removal of vdev 0 copied 10.7G in 0h1m

Check new size

zpool list rpool

Shutdown and remove the drives and reboot to make sure everything is working as intended
Remove the old drives from proxmox-boot-tool

proxmox-boot-tool status
proxmox-boot-tool clean
proxmox-boot-tool status

Remove all data from old drives

put the drives back in
wipe them using your favorite tool (eg. shred or dd)

dd if=/dev/zero of=/dev/sdX -bs=1M status=progress
dd if=/dev/urandom of=/dev/sdX -bs=1M status=progress

Conclusion

I saw a 7-8W drop in idle power usage and an increase in responsiveness.

Appendix

Sources

10 June 2024

r730xd Server Part 3: Software

I will be using Proxmox Helper Scripts (https://tteck.github.io/Proxmox/ or https://Helper-Scripts.com) to help configure the different LXCs and VMs that I want.

Disable nag screen

As I was tired of having to confirm that I didn't have a subscription, I ripped these commands from Proxmox VE Tools -> Proxmox VE Post Install (https://raw.githubusercontent.com/tteck/Proxmox/main/misc/post-pve-install.sh) and ran on the Shell command line

echo "DPkg::Post-Invoke { \"dpkg -V proxmox-widget-toolkit | grep -q '/proxmoxlib\.js$'; if [ \$? -eq 1 ]; then { echo 'Removing subscription nag from UI...'; sed -i '/.*data\.status.*{/{s/\!//;s/active/NoMoreNagging/}' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js; }; fi\"; };" >/etc/apt/apt.conf.d/no-nag-script
apt --reinstall install proxmox-widget-toolkit

Setup Users

My philosophy was to put all actual users in the 1000s and system users in the 2000s

user1

useradd user1
usermod -g users user1
adduser user1 user1
id user1

user2

useradd user2
usermod -g users user2
adduser user2 user2
id user2

mythtv

groupadd -g 2001 mythtv
useradd -u 2001 -g 2001 mythtv

nginx

groupadd -g 2002 nginx
useradd -u 2002 -g 2002 nginx

edit /etc/subuid and add

root:1000:1000
root:2000:1000

edit /etc/subgid and add

root:100:1
root:1000:1000
root:2000:1000

Import ZFS

zpool import storage
I decided not to map the drive in Proxmox, but if you wanted to you would do that here

proxmox -> Datacenter -> Storage -> Add -> ZFS

Fix Directory/File Permissions

/storage/mythtv

cd /storage/mythtv
ls -al
find ./ -user <current owner> -print0 | xargs -0 chown -h mythtv
find ./ -group <current group> -print0 | xargs -0 chgrp -h mythtv

/storage/containers/mythtv

cd /storage/containers/mythtv
ls -al
find ./ -user <current owner> -print0 | xargs -0 chown -h mythtv
find ./ -group <current group> -print0 | xargs -0 chgrp -h mythtv

/storage/containers/webserver

cd /storage/containers
chown -R nginx webserver
chgrp -R nginx webserver

Create a Docker LXC

Now I needed a Docker LXC to run my webserver and MythTV

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/docker.sh)"
I then edited the config to set a static IP and change the hostname

docker -> Network -> net0 -> Edit
docker -> DNS -> Hostname -> Edit

Add users

groupadd -g 2000 nas
useradd -u 2000 -g 2000 nas

mythtv

groupadd -g 2001 mythtv
useradd -u 2001 -g 2001 mythtv

nginx

groupadd -g 2002 nginx
useradd -u 2002 -g 2002 nginx

MythTV

Edit /storage/containers/mythtv/docker-compose.yml

Change the User Ids and Groups Ids to 2001

Test

docker compose up -d

If it looks like API port changed from 6544 to 6744, then you need to fix the IPs
Check pin and update backend ip

apt-get install default-mysql-client
mysql -p -h 127.0.0.1 -P 3306 mythconverg

select * from settings where value like '%pin%'
update settings set data = '192.168.1.31' where data = '192.168.1.11' ;
quit;

Restart MythTV and Test

docker compose down && docker compose up -d

This time I put a copy of docker-mythtv.service in /storage/container/mythtv so I can easily copy it to /etc/systemd/system/ in the future

make sure to change `docker-compose` to `docker compose`

docker compose down
systemctl enable docker-mythtv
systemctl start docker-mythtv

Webserver

Test

docker compose up -d
docker compose down

This time I put a copy of docker-webserver.service, certbot.service, and certbot.timer in /storage/container/webserver/systemd so I can easily copy it to /etc/systemd/system/ in the future

make sure to change `docker-compose` to `docker compose`

cp certbot.service certbot.timer docker-webserver.server /etc/systemd/system/
systemctl enable docker-webserver
systemctl start docker-webserver
systemctl enable certbot.timer

Create a Debian LXC for File Sharing

following:

https://www.youtube.com/watch?v=Hu3t8pcq8O0

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/debian.sh)"
Add users/groups

<container> -> Console
user1

useradd user1
usermod -g users user1
adduser user1 user1
id user2

user2

useradd user2
usermod -g users user2
adduser user2 user2
id user2

groupadd -g 2000 nas
useradd nas -u 2000 -g 2000 -m -s /bin/bash
adduser nas sudo
passwd nas

shutdown the container
Set a static IP and change the hostname

<container> -> Network -> net0 -> Edit
<container> -> DNS -> Hostname -> Edit

Add more compute/memory (2 cores/1024MB)

<container> -> Resources -> Cores -> Edit
<container> -> Resources -> Memory -> Edit

Add our storage

edit /etc/pve/lxc/<container id>.conf
add a line for each of your datasets like the below examples:

mp0: /storage/folder1/dataset1,mp=/storage/folder1/dataset1
mp1: /storage/folder1/dataset2,mp=/storage/folder1/dataset2
mp2: /storage/dataset3,mp=/storage/dataset3

Map the users

Notes

/etc/subuid and /etc/subgid need to specify the user starting the lxc container (root)
/etc/pve/lxc/<container id>.conf needs to map all ids and not just the ones you want to remap

edit /etc/pve/lxc/<container id>.conf add these lines

lxc.idmap: u 0 100000 1000
lxc.idmap: u 1000 1000 1000
lxc.idmap: u 2000 102000 63535
lxc.idmap: g 0 100000 100
lxc.idmap: g 100 100 1
lxc.idmap: g 101 100101 899
lxc.idmap: g 1000 1000 1000
lxc.idmap: g 2000 102000 63535

~~Change the drive permissions~~

I couldn't get the container to boot with trying to remap root so don't do this step
~~cd /rpool/data/subvol-<container id>-disk-0~~
~~find ./ -user 100000 -print0 | xargs -0 chown -h 2000~~
~~find ./ -group 100000 -print0 | xargs -0 chgrp -h 2000~~

Start the container
Install cockpit

apt install cockpit --no-install-recommends
wget https://github.com/45Drives/cockpit-file-sharing/releases/download/v3.3.7/cockpit-file-sharing_3.3.7-1focal_all.deb
wget https://github.com/45Drives/cockpit-navigator/releases/download/v0.5.10/cockpit-navigator_0.5.10-1focal_all.deb
wget https://github.com/45Drives/cockpit-identities/releases/download/v0.1.12/cockpit-identities_0.1.12-1focal_all.deb
apt install ./*.deb
rm *.deb

Configure cockpit

https://192.168.X.X:9090
use nas to login
enable administrative access
Identities

Set a Samba password for each of the users

File Sharing

Click "Fix Now"
Global Settings

Toggle Global MacOS Shares
Add `allow insecure wide links = yes` to Advanced
Apply

Add your shares
I used the following to ensure all users can access others files in advanced

create mask = 0664
force create mode = 0664
directory mask = 0775
force directory mode = 0775

I used the following to be able to follow symlinks

follow symlinks = yes
wide links = yes

Create a Ubuntu VM for VNC and Handbrake

I have read that x264 sees reduced performance with over 6 threads, so I gave the VM 12 virtual cores since I want to be able to process 2 discs at a time.