r730xd Server Part 1: Initial Build

In order to support more than the 2 hard drives I could in the TS140, I needed a new system. Here were my requirements:

• price < $500
• reasonable noise level
• more than 4 cores
• ECC RAM (preferably at least 128GB)
• 8 or more spots for hard drives

I was looking at some older SuperMicro and Dell systems. I read several posts about the power supplies being louder on the SuperMicro so I decided to go with the Dell r730xd that supports 12 x 3.5" drives in the front and 2 x 2.5" drives in the rear.

Hardware

• r730xd used - $350 + tax
• 2 x e5-2670v3 12 cores each
• 128 GB ECC DDR4 (8 x 16GB)
• PERC H730p Mini raid controller
• 1 x 2.5" caddie with 1.2TB 10K SAS drive
• iDRAC 8 Enterprise license
• 12 x 3.5" used caddies - $60 + tax
• SAS or SATA caddie both work for holding sata drives
• 1 x 2.5" caddie with 1.2 TB 10K SAS drive - $24 + tax
• this is a second drive for a boot mirror
• total: $434 + tax

TLDR; Pros/Cons

• Pros
• Many many cores
• Loads of RAM with expandability
• Thermal solution works quite well even under full load with fans at 6-8% (2640-2880 rpm) on auto keeping the processors around 77C
• iDRAC is quite useful especially the HTML5 virtual console (requires Enterprise license)
• Cons
• No hardware reset for iDRAC
• Somewhere in the middle
• idle power usage of 94-96W (with a single 2.5" drive) is higher than I would like
• Fans @ 4% (2280 rpm)
• 18W for iDRAC even when powered down
• installing hard drive
• when installing an older drive (250GB WD) fans spun up to 40% (7320 rpm), this may have been because it was unable to read temp or was failing
• installing a 3TB WD Red drive caused the fans to go to 6% (2680 rpm)
• Fan noise
• 45-50% (8880-9840 rpm) is noticeable with the basement door closed
• 30% (6480 rpm) is noticeable with the basement door open
• 15-20% (4080-4920 rpm) about as loud as a running fridge
• 10% (3240 rpm) a low constant hum
• 6% (2640 rpm) quieter than seeking dvd drives

BIOS/iDRAC reset and setup

First I did a BIOS reset by moving the jumper. Unfortunately, this did not reset the iDRAC, and there does not seem to be a way to do this with hardware. For this you will need a VGA monitor and keyboard. I followed this video to do it: https://www.youtube.com/watch?v=R5pYfqDtzQw

Reset BIOS

• F2 for System Setup
• System BIOS
• Default
• Finish and Confirm
• Reboot

BIOS Settings to change from default

• Boot Settings -> Boot Mode -> UEFI
• System Profile Settings -> System Profile -> Performance per Watt (OS)
• Finish and Confirm
• Reboot

Reset iDRAC

• F2 for System Setup
• iDRAC Settings
• Reset iDRAC configurations to defaults
• Finish and Confirm
• Reboot

Then I was able to configure iDRAC from its default 192.168.0.120 to an unused IP on my network.

• F2 for System Setup
• iDRAC Settings
• Network
• Update Static IP Address (and Subnet/Gateway if applicable)
• Back
• Finish and Confirm
• Reboot

Memory Testing

While memtest86 isn't the best memory tester available, it is free and doesn't require an OS to be installed. I used version 10.7, which I downloaded the ISO for loading onto my Ventoy USB from here: https://www.memtest86.com/downloads/memtest86-iso.zip

It took about 24 hours to do all 4 passes of the test patterns.

Update iDRAC

Here is where I downloaded the iDRAC update from: https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/drivers. You should download the version for windows for uploading through the Web UI. Then apply the updates from the iDRAC interface: Login to iDRAC -> iDRAC Settings -> Update and Rollback -> upload

At first I download the win64 version of the latest available (2.85.85.85), but the update would fail due to it not validating. Here is the update path that I went with: 2.61.60.60 -> 2.63.60.61 (win32) -> 2.75.100.75 (win32) -> 2.85.85.85 (win64)

After the first update the iDRAC appeared to be unresponsive, so I let it sit for ~20 minutes and it was still unresponsive, however I could ping it. At this point I held in the "i" button for ~20s to restart the iDRAC. This brought it back to life.

Here is a video that I found helpful: https://www.youtube.com/watch?v=dea-PA1na0c

Update the rest of the firmware

I always update the BIOS separately

Update BIOS/UEFI:

• Login to iDRAC -> iDRAC Settings -> Update and Rollback -> HTTPS
• HTTPS Address: downloads.dell.com
• Check for Updates
• Select the BIOS update and click "Install and Reboot"
• The system will then reboot and install the new BIOS

Update Raid/Network:

• Login to iDRAC -> iDRAC Settings -> Update and Rollback -> HTTPS
• HTTPS Address: downloads.dell.com
• Check for Updates
• Select the Raid and Network update and click "Install and Reboot"
• The system will then reboot and install the new BIOS

Set Raid card to HBA mode

This will pass the drives directly to the OS to manage the software raid.

Login to iDRAC -> Storage -> Controllers -> Setup -> Controller Mode -> Action -> HBA

Proxmox install

I downloaded Proxmox 8.1 from: https://www.proxmox.com/en/downloads

I was using a Ventoy USB to do my Proxmox 8.1 install, but it was giving me an error. However, I found that you need to use Ventoy >= 1.0.97 (https://github.com/ventoy/Ventoy/releases/tag/v1.0.97). So I updated my Ventoy USB and then the install proceeded.

During the setup here are some of the values that I chose:

• Target -> Options
• If you only have 1/2 of your boot drives
• ZFS (RAID0)
• Harddisk 0 - 1.09TB
• Harddisk 1 - Do not use
• If you have both go ahead and setup the mirror
• ZFS (RAID1)
• Harddisk 0 - 1.09TB
• Harddisk 1 - 1.09TB
• Timezone: America/New_York
• Password and Email
• Management
• Make sure to select the correct management interface (easier if network is hooked up before starting setup)
• Hostname: r730xd.home.arpa
• IP Address: pick a free IP
• Gateway/DNS

After the setup, login to the web ui: https://<IP>:8006

• Disable Proxmox Enterprise repos as they require a license
• Click on the hostname (r730xd)
• Remove the subscription update repository
• Updates -> Repositories
• Highlight "pve pve-enterprise" and click "Disable"
• Highlight "ceph-quincy enterprise" and click "Disable"
• Add the No Subscript update repository
• per https://pve.proxmox.com/pve-docs/chapter-sysadmin.html for NON-PRODUCTION updates
• Click "Add"
• Select "No-Subscription"
• Click "Add"
• Click "Add"
• Select "Ceph Quincy No-Subscription"
• Click "Add"
• Update your install
• Updates
• Click "Refresh"
• Close the window when it is done
• Click "Upgrade"
• Type "y" and enter

Add the rpool mirror (if not configured at install)

• Determine if using grub or systemd-boot
• proxmox-boot-tool status
• efibootmgr -v
• Copy the partition table over (removing disk label-id and uuids so sfdisk generates new ones)
• sfdisk -d /dev/sda > part_table
• sed -e '/^label-id:/d' -e 's/,\s*uuid=[-0-9A-F]*//g' part_table | sfdisk /dev/sdb
• Copy the bios boot
• dd if=/dev/sda1 of=/dev/sdb1
• Prepare EFI partition
• dd if=/dev/sda2 of=/dev/sdb2 Don't do this as it keeps the UUID
• OR
• proxmox-boot-tool format /dev/sdb2
• Initialize the EFT partition
• proxmox-boot-tool init /dev/sdb2
• OR if using grub
• proxmox-boot-tool init /dev/sdb2 grub
• Get current zpool drive
• zpool status
• Add the mirror
• zpool attach rpool scsi-XXXXXXXXX-part3 /dev/disk/by-id/scsi-YYYYYYYYY-part3
• ZFS will then resilver the drive, to check if done run
• zpool status

intel_pstate

Trying to lower the idle power usage, I wanted to try a different cpu governor driver.

• check whether it is intel_pstate or intel_cpufreq
• apt install cpufrequtils
• cpufreq-info | head -n 13
• Determine if using grub or systemd-boot
• proxmox-boot-tool status
• efibootmgr -v
• to enable intel_pstate for grub boot:
• edit /etc/default/grub
• add intel_pstate=active to GRUB_CMDLINE_LINUX_DEFAULT
• example: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=active"
• proxmox-boot-tool refresh
• wait for grub to finish updating and then reboot
• to enable intel_pstate for systemd-boot
• edit /etc/kernel/cmdline
• add intel_pstate=active to the end
• proxmox-boot-tool refresh
• wait for it to update and then reboot
• ensure it is now intel_pstate
• cpufreq-info | head -n 13
• unfortunately this did not change idle power usage

Fan Information

Here are the fan values from my r730xd

•  4% -> 2280 rpm -> 10 cfm
•  6% -> 2640 rpm -> 12 cfm
•  9% -> 3000 rpm -> 18 cfm
• 12% -> 3600 rpm -> 21 cfm
• 15% -> 4080 rpm -> 24 cfm
• 20% -> 4920 rpm -> 30 cfm
• 25% -> 5640 rpm -> 36 cfm
• 30% -> 6480 rpm -> 41 cfm
• 40% -> 7320 rpm -> 53 cfm
• 50% -> 8160 rpm -> 64 cfm

Appendix

Sources

• copying part tables
• https://forum.proxmox.com/threads/how-do-i-know-if-proxmox-is-booting-with-grub-or-systemd-boot.67020/
• https://forum.proxmox.com/threads/creating-a-zfs-boot-mirror.119027/
• https://www.youtube.com/watch?v=m4zwJvzQ65s
• intel_pstate: 
• https://bbs.archlinux.org/viewtopic.php?id=260361
• https://forum.proxmox.com/threads/edit-systemd-boot.136136/

Checking if two video files are the same aside from metadata

As I wanted to make sure that two video files were identical aside from metadata, I decided to write a script to help me. The script will create a hash from the audio and video data without the header data.

Caveats:

• There is a chance for hash collisions, so double check the results manually before deleting
• This script doesn't flag videos as duplicates if they are the same video but have different resolution, bitrate, audio data, etc

Requirements

• ffmpeg
• md5sum

Shell Version

Initial shell script to compare to 2 files: 

#!/bin/sh

file1="$1"

file2="$2"

flags="-fflags +bitexact -flags:v +bitexact -flags:a +bitexact -c copy -f matroska"

file1hash=$(ffmpeg -i "$file1" $flags -c copy -f matroska -loglevel error - | md5sum | cut -f1 -d" ")

file2hash=$(ffmpeg -i "$file2" $flags -c copy -f matroska -loglevel error - | md5sum | cut -f1 -d" ")

echo "$file1hash $file1"

echo "$file2hash $file2"

Example Shell

$./diffvideo.sh file1.m4v file2.m4v

cb31XXXXXXXXXXXXXXXXXXXXXXXXXXXX file1.m4v

cb31XXXXXXXXXXXXXXXXXXXXXXXXXXXX file2.m4v

Python Version

Expanded python script to compare more files:

#!/usr/bin/env python3

import argparse

import shlex

import subprocess

def setup_cli():

    parser = argparse.ArgumentParser(

        prog='',

        description='',

        epilog='',

    )

    parser.add_argument('filenames', nargs='*')

    return parser

def check_files(filenames):

    hashes = {'not_a_video': []}

    for file in filenames:

        flags = '-fflags +bitexact -flags:v +bitexact -flags:a +bitexact'

        cmd = f'ffmpeg -i "{file}" {flags} -c copy -f matroska -'

        ff_proc = subprocess.run(shlex.split(cmd), capture_output=True)

        if ff_proc.returncode != 0:

            hashes['not_a_video'].append(file)

            continue

        hash_proc = subprocess.run('md5sum', capture_output=True, input=ff_proc.stdout)

        filehash = hash_proc.stdout.decode().split()[0]

        if filehash not in hashes:

            hashes[filehash] = []

        hashes[filehash].append(file)

    return hashes

def print_results(hashes):

    not_a_video = hashes.pop('not_a_video')

    singles = {}

    dupes = {}

    for key, value in hashes.items():

        if len(value) > 1:

            dupes[key] = value

        else:

            singles[key] = value

    if not_a_video:

        print('\nNot a video:')

    for each in not_a_video:

        print(f'    {each}')

    if dupes:

        print('\nDuplicates found:')

    for key, value in dupes.items():

        print(f'    {key}')

        for v in value:

            print(f'        {v}')

    if singles:

        print('\nNo Duplicates for these files:')

    for key, value in singles.items():

        print(f'    {value[0]}')

if __name__ == "__main__":

    parser = setup_cli()

    args = parser.parse_args()

    hashes = check_files(args.filenames)

    print_results(hashes)

Example Python

$./diffvideo.py *

Not a video:

    test.txt

    file.docx

Duplicates found:

    cb31XXXXXXXXXXXXXXXXXXXXXXXXXXXX

        file1.m4v

        file2.m4v

    ab76XXXXXXXXXXXXXXXXXXXXXXXXXXXX

        file5.m4v

        file6.m4v

No duplicates for these files:

    file3.m4v

    file4.m4v

Appendix

Sources

• https://stackoverflow.com/questions/38964861/how-to-remove-all-metadata-using-ffmpeg
• https://stackoverflow.com/questions/46567146/how-to-get-stdout-from-ffmpeg-without-format