Cleaning up old vmlinuz and initrd files in /boot

 I noticed that updates were taking a long time on the kernel step. So I expanded what it was doing and it was attempting to build really old versions (3.13)

Purge from apt-get

• Figure out which package to remove
• dpkg -S /boot/vmlinuz-3.13.0-36-generic
• Remove the package
• sudo apt-get purge linux-image-3.13.0-36-generic
• If you have many to remove, you can use bash expansion like so:
• sudo apt-get purge linux-image-3.13.0-{36,37,39}-generic
• Regenerate initrd
• sudo update-initramfs -u -k all
• Update grub
• sudo update-grub
• If initramfs is still generating stuff for old kernel, you can remove like so
• sudo update-initramfs -d -k 3.13.0-36-generic
• Note: bash expansion does not work for this command

Sources

• https://askubuntu.com/questions/912473/can-i-delete-old-vmlinuz-in-boot-manually
• https://askubuntu.com/questions/1001285/why-are-old-initrd-files-of-uninstalled-kernels-filling-up-boot-partition

Replacing drives in ZFS

 As my two 3TB WD Red drives (purchased before the SMR controversy) in Raid 1 were starting to fill up and the price of bigger drives came down, it was time for an upgrade. I purchased two 6TB WD Red Plus drives to replace them (~$105 each before tax). I went with this particular drive and size because of the speed (5640 rpm) to keep the drives quiet and cool. Most NAS drives are going to the higher rpm of 7200.

Checking the drives

I used an external enclosure to test each of the drives before adding them to the ZFS Pool

• Determine the drive
• For these examples I used /dev/sdX, but you need to use the device letter applicable to you
• ls /dev/sd*
• Check initial SMART attributes
• sudo smartctl -A /dev/sdX
• SMART short test
• sudo smartctl --test=short /dev/sdX
• Check SMART test status
• sudo smartctl -c /dev/sdX
• Check SMART attributes
• sudo smartctl -A /dev/sdX
• Write Zeros to the entire disk
• Warning: This will erase the entire disk and you will lose anything stored on it
• sudo dd if=/dev/zero of=/dev/sdX bs=1M status=progress
• Check SMART attributes
• sudo smartctl -A /dev/sdX
• SMART extended test
• I had problems with this test being stuck at 90% remaining due to the drive stopping spinning
• The solution it to make sure the drive is spinning and then kicking off the test and a command to keep the drive active
• sudo dd if=/dev/sdX of=/dev/null count=1  # Read one 512 byte block from drive
• sudo smartctl --test=long /dev/sdX  # Kick off the test
• sudo watch -d -n 60 smartctl -a /dev/sdX  # Make sure the drive doesn't spin down. Will also show test status
• Power Off the drive
• sudo udisksctl power-off -b /dev/sdX
• Disconnect the drive
• Repeat for the next drive

Adding the drives to the pool

At this point I hooked both drives up to the SATA connectors inside the server, but had to disconnect the front panel USB to access the open SATA port and disconnect the DVD drive. Now it was time to add them to the pool to get data replicated to them.

• Get current status
• sudo zpool status
• Get the drive ids
• ls -l /dev/disk/by-id/
• Attach them to a current drive to put them in the same mirror
• Run this command with the values substituted for both drives:
• sudo zpool attach [poolname] [original drive to be mirrored] [new drive]
• If you get an error like: devices have different sector alignment
• than add: -o ashift=9 
• There are performance penalties for running a 4K drive as a 512B one, but supposedly there are storage size advantages
• Update 2023-12-10: Apparently it is a quite large performance difference: 35-45MB/s vs 70-72MB/s
• Here is what I ran (serial 1 and 2 are the existing drives while 3 and 4 are the new ones:
• sudo zpool attach storage /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#1 /dev/disk/by-id/ata-WDC_WD60EFZX-68B3FN0_WD-SERIAL#3 -o ashift=9
• sudo zpool attach storage /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#2 /dev/disk/by-id/ata-WDC_WD60EFZX-68B3FN0_WD-SERIAL#4 -o ashift=9
• Check on the status:
• sudo zpool status
• Wait for the resilvering to complete
• Estimated time was ~33 hours
• Actual runtime was ~18 hours copying around 2TB of data
• Optional scrub to verify all data
• sudo zpool scrub [POOLNAME]
• Wait for it to finish
• Estimated time was ~13 hours
• Actual time was ~10 hours
  

Remove the old drives

• Remove the old disks from the pool:
• sudo zpool detach [POOLNAME] [DISKNAME]
• What I ran:
• sudo zpool detach storage /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#1
• sudo zpool detach storage /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#2
• Check on the status
• sudo zpool status
• Shutdown the server and remove the drives

Expand the Quotas

• Check current usage and quotas
• df -k /storage/mythtv /storage/.encrypted
• sudo zfs get quota storage/mythtv
• sudo zfs get quota storage/.encrypted
• Set new quotas, total storage available was 5.45T
• sudo zfs set quota=1.5T storage/mythtv
• sudo zfs set quota=3.9T storage/.encrypted
• While editing my zfs containers, I decided to turn off compression since i wasn't seeing any gains
• sudo zfs set compression=off storage/.encrypted

Sources

• Drive Check
• Initial inspiration: https://superuser.com/questions/1252662/initial-testing-of-new-hard-drives-in-linux
• Keeping the drive spinning: https://superuser.com/questions/766943/smart-test-never-finishes
• Power Off the drive: https://askubuntu.com/questions/532586/what-is-the-command-line-equivalent-of-safely-remove-drive
• Adding to zpool
• https://www.sotechdesign.com.au/how-to-add-a-drive-to-a-zfs-mirror/
• https://superuser.com/questions/1265328/how-to-create-zfs-mirror-with-disks-of-different-sector-sizes
• Performance difference of ashift=9 vs ashift=12
• https://louwrentius.com/zfs-performance-and-capacity-impact-of-ashift9-on-4k-sector-drives.html
• Removing from zpool
• https://www.truenas.com/community/threads/remove-disk-from-mirror-in-freenas-11.56494/

Let's Encrypt SSL Certificate for Dynamic DNS Host

As I was tired of having to add an exception when wanting to visit my personal website, I decided it was time to use Let's Encrypt to get a signed SSL certificate. Luckily it was easier than I thought. Here are the simple steps that I followed.

1. Add the ppa repository
• sudo add-apt-repository ppa:certbot/certbot
2. Install the cli tool
• sudo apt install python-certbot-apache
3. Run the cli tool
• sudo certbot --apache -d <dynamic dns host name>
4. Answer the questions that included:
• Email Address
• Select which conf file to add to (For me it was /etc/apache2/sites-available/default-ssl.conf)
5. Check that my SSL certificate was now signed

TODO: Automate the renewal

ThinkServer TS140 Build

As my old server is 5 years old (2008-12) and been showing its age, I decided it was time for an upgrade.

Hardware:

1x Lenovo  ThinkServer TS140 70A4001LUX  -  $399.99 - $50 rebate (newegg)

• Intel Xeon E3-1225v3 3.2GHz
• 4GB DDR3 1600
• No Hard Drive
• Gigabit Ethernet 

2x Western Digital Red 3 TB  -  $114.99 a piece (newegg)
1x Old Western Digital Caviar 250 GB w/ sata cable  -  Already had
1x 18 inch Monoprice Sata III cable with 90 degree angle plug  -  $2.66 (amazon)
1x Trendnet USB to PS/2 adapter (TU-PS2)  -  $8.99 (amazon)
1x Small zip tie  -  Already had

Software:

Xubuntu 14.04 LTS 64bit  -  Free (xubuntu.org/getxubuntu/)

Total Cost: $591.62 + tax

Hardware Installation:

1. Opened up the case
1. Removed Thumb screws (had to use screwdriver to get it started)
2. Pressed in the side release button
3. Removed side panel
2. Installed the 250GB primary drive (in what Lenovo calls the tertiary drive slot)
1. Pressed the quick release button
2. Pushed the 5.25 to 3.5 adapter out from the inside
3. Be careful as the fan power cord is not long!
4. Placed the hard drive in the adapter
5. Flipped over while supporting hard drive
6. Aligned screw holes and fastened screws (make sure to use included screws as they are longer due to anti-vibration gaskets)
7. Pushed adapter back into place and assured that quick release snapped back into place
8. Attached Sata
1. Removed front USB motherboard connector
2. Moved existing Sata cable from Sata0 to Sata1
3. Reattached front USB motherboard connector
4. Attached Sata cable to Sata0 ran thru cord fastener on motherboard side and attached to HD
9. Attached Power
1. Clipped zip tie holding power cable in place
2. Un-did loop in power cable so it could reach HD
3. Attached power cable to HD
4. Zip tied all cables back into place
3. Installed the 2x 3TB drives
1. Pulled out the drive caddy
2. Installed pegs into one side of the HD
3. Stretched caddy so that pegs would go into other side of the HD
4. Pushed HD + caddy back into place
5. Attached sata (Sata1 and Sata2) and power cable

Testing:

I wanted to do an extended drive test on all the drives to ensure they were in good working order before proceeding with my software install. I used the Western Digital DOS diagnostic tool (wdc.com). It took about 7 hours for each of the 3TB drives and just over an hour for the 250GB drive.

Software Installation:

1. First I created a linux live USB from the ISO
2. Booted from the USB in Try Xubuntu mode
3. As I was recycling a drive from an old FakeRaid array, I had to delete the raid metadata. I opened up a terminal window and ran:
• sudo dmraid -E -r /dev/sdX
• where X is the letter of the old raid array drive
4. Ran the install Xubuntu icon on the desktop
5. Followed the prompts to install to the 250GB drive
• Used advanced partition setup:
• /boot: 537MB
• /: 100GB
• /home: 139 GB
• swap: 8GB
6. Remove USB and Reboot 
7. Installed Updates
• sudo update-manager
8. Reboot 
9. Made sure Red Drives had WD Load/Unload update installed
• http://support.wdc.com/product/download.asp?groupid=619&sid=201&lang=en
10. Setup the ZFS mirrored raid 
1. sudo apt-get install zfs-fuse
2. ls /dev/disk/by-id/
3. sudo zpool create storage mirror /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#1 /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-SERIAL#2
4. sudo zpool status

Thoughts:

• The server is dead quiet which is one of the major reasons I bought it
• With the Trendnet for some reason the scroll on the mouse causes the mouse to jump instead of scrolling, but as I plan to use this as a mainly server this is a small issue

Server Build

Here are the components to my server build from 2009-12:

1. Dual Xeon Server - $209.99 from geeks.com




• 1x Intel Server Board SE7501BR2


• 2x 2.4GHz Xeon Processors


• 2x 512MB ECC DDR


• 3x 36GB SCSI Hard Drives in Raid


• 1x 40GB IDE Hard Drive - Removed


• 1x DVD Drive


• 1x CD-RW Drive


• 1x Floppy Drive




2. 2x Micron 1GB DDR PC2100 ECC memory chips - $46 for both from geeks.com


3. 1x HP Adaptec 2610SA PCI-X Raid Card - $79 from ebay.com


4. 2x WD Green 1TB 16MB SATA 3Gbps - $209.98 for both from newegg.com


5. 2x SATA Cables - $5.98 for both from newegg.com


6. 1x Powercolor 2400 Pro PCI video car - $36.99 from newegg.com


7. Shipping: $44 + $11.45 + $5.99

Total: 649.38