In order to support more than the 2 hard drives I could in the TS140, I needed a new system. Here were my requirements:
- price < $500
- reasonable noise level
- more than 4 cores
- ECC RAM (preferably at least 128GB)
- 8 or more spots for hard drives
I was looking at some older SuperMicro and Dell systems. I read several posts about the power supplies being louder on the SuperMicro so I decided to go with the Dell r730xd that supports 12 x 3.5" drives in the front and 2 x 2.5" drives in the rear.
Hardware
- r730xd used - $350 + tax
- 2 x e5-2670v3 12 cores each
- 128 GB ECC DDR4 (8 x 16GB)
- PERC H730p Mini raid controller
- 1 x 2.5" caddie with 1.2TB 10K SAS drive
- iDRAC 8 Enterprise license
- 12 x 3.5" used caddies - $60 + tax
- SAS or SATA caddie both work for holding sata drives
- 1 x 2.5" caddie with 1.2 TB 10K SAS drive - $24 + tax
- this is a second drive for a boot mirror
- total: $434 + tax
TLDR; Pros/Cons
- Pros
- Many many cores
- Loads of RAM with expandability
- Thermal solution works quite well even under full load with fans at 6-8% (2640-2880 rpm) on auto keeping the processors around 77C
- iDRAC is quite useful especially the HTML5 virtual console (requires Enterprise license)
- Cons
- No hardware reset for iDRAC
- Somewhere in the middle
- idle power usage of 94-96W (with a single 2.5" drive) is higher than I would like
- Fans @ 4% (2280 rpm)
- 18W for iDRAC even when powered down
- installing hard drive
- when installing an older drive (250GB WD) fans spun up to 40% (7320 rpm), this may have been because it was unable to read temp or was failing
- installing a 3TB WD Red drive caused the fans to go to 6% (2680 rpm)
- Fan noise
- 45-50% (8880-9840 rpm) is noticeable with the basement door closed
- 30% (6480 rpm) is noticeable with the basement door open
- 15-20% (4080-4920 rpm) about as loud as a running fridge
- 10% (3240 rpm) a low constant hum
- 6% (2640 rpm) quieter than seeking dvd drives
BIOS/iDRAC reset and setup
First I did a BIOS reset by moving the jumper. Unfortunately, this did not reset the iDRAC, and there does not seem to be a way to do this with hardware. For this you will need a VGA monitor and keyboard. I followed this video to do it:
https://www.youtube.com/watch?v=R5pYfqDtzQwReset BIOS
- F2 for System Setup
- System BIOS
- Default
- Finish and Confirm
- Reboot
BIOS Settings to change from default
- Boot Settings -> Boot Mode -> UEFI
- System Profile Settings -> System Profile -> Performance per Watt (OS)
- Integrated Devices -> USB 3.0 Setting -> Enable (Added 2024-06-07)
- Finish and Confirm
- Reboot
Reset iDRAC
- F2 for System Setup
- iDRAC Settings
- Reset iDRAC configurations to defaults
- Finish and Confirm
- Reboot
Then I was able to configure iDRAC from its default 192.168.0.120 to an unused IP on my network.
- F2 for System Setup
- iDRAC Settings
- Network
- Update Static IP Address (and Subnet/Gateway if applicable)
- Back
- Finish and Confirm
- Reboot
Memory Testing
While memtest86 isn't the best memory tester available, it is free and doesn't require an OS to be installed. I used version 10.7, which I downloaded the ISO for loading onto my Ventoy USB from here: https://www.memtest86.com/downloads/memtest86-iso.zip
It took about 24 hours to do all 4 passes of the test patterns.
Update iDRAC
Here is where I downloaded the iDRAC update from: https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/drivers. You should download the version for windows for uploading through the Web UI. Then apply the updates from the iDRAC interface: Login to iDRAC -> iDRAC Settings -> Update and Rollback -> upload
At first I download the win64 version of the latest available (2.85.85.85), but the update would fail due to it not validating. Here is the update path that I went with: 2.61.60.60 -> 2.63.60.61 (win32) -> 2.75.100.75 (win32) -> 2.85.85.85 (win64)
After the first update the iDRAC appeared to be unresponsive, so I let it sit for ~20 minutes and it was still unresponsive, however I could ping it. At this point I held in the "i" button for ~20s to restart the iDRAC. This brought it back to life.
Here is a video that I found helpful: https://www.youtube.com/watch?v=dea-PA1na0c
Update the rest of the firmware
I always update the BIOS separately
Update BIOS/UEFI:
- Login to iDRAC -> iDRAC Settings -> Update and Rollback -> HTTPS
- HTTPS Address: downloads.dell.com
- Check for Updates
- Select the BIOS update and click "Install and Reboot"
- The system will then reboot and install the new BIOS
Update Raid/Network:
- Login to iDRAC -> iDRAC Settings -> Update and Rollback -> HTTPS
- HTTPS Address: downloads.dell.com
- Check for Updates
- Select the Raid and Network update and click "Install and Reboot"
- The system will then reboot and install the new BIOS
Set Raid card to HBA mode
This will pass the drives directly to the OS to manage the software raid.
Login to iDRAC -> Storage -> Controllers -> Setup -> Controller Mode -> Action -> HBA
Proxmox install
I downloaded Proxmox 8.1 from: https://www.proxmox.com/en/downloads
I was using a Ventoy USB to do my Proxmox 8.1 install, but it was giving me an error. However, I found that you need to use Ventoy >= 1.0.97 (https://github.com/ventoy/Ventoy/releases/tag/v1.0.97). So I updated my Ventoy USB and then the install proceeded.
During the setup here are some of the values that I chose:
- Target -> Options
- If you only have 1/2 of your boot drives
- ZFS (RAID0)
- Harddisk 0 - 1.09TB
- Harddisk 1 - Do not use
- If you have both go ahead and setup the mirror
- ZFS (RAID1)
- Harddisk 0 - 1.09TB
- Harddisk 1 - 1.09TB
- Timezone: America/New_York
- Password and Email
- Management
- Make sure to select the correct management interface (easier if network is hooked up before starting setup)
- Hostname: r730xd.home.arpa
- IP Address: pick a free IP
- Gateway/DNS
After the setup, login to the web ui: https://<IP>:8006
- Disable Proxmox Enterprise repos as they require a license
- Click on the hostname (r730xd)
- Remove the subscription update repository
- Updates -> Repositories
- Highlight "pve pve-enterprise" and click "Disable"
- Highlight "ceph-quincy enterprise" and click "Disable"
- Add the No Subscript update repository
- Update your install
- Updates
- Click "Refresh"
- Close the window when it is done
- Click "Upgrade"
- Type "y" and enter
Add the rpool mirror (if not configured at install)
- Determine if using grub or systemd-boot
- proxmox-boot-tool status
- efibootmgr -v
- Copy the partition table over (removing disk label-id and uuids so sfdisk generates new ones)
- sfdisk -d /dev/sda > part_table
- sed -e '/^label-id:/d' -e 's/,\s*uuid=[-0-9A-F]*//g' part_table | sfdisk /dev/sdb
- Copy the bios boot
- dd if=/dev/sda1 of=/dev/sdb1
- Prepare EFI partition
dd if=/dev/sda2 of=/dev/sdb2 Don't do this as it keeps the UUID- OR
- proxmox-boot-tool format /dev/sdb2
- Initialize the EFT partition
- proxmox-boot-tool init /dev/sdb2
- OR if using grub
- proxmox-boot-tool init /dev/sdb2 grub
- Get current zpool drive
- Add the mirror
- zpool attach rpool scsi-XXXXXXXXX-part3 /dev/disk/by-id/scsi-YYYYYYYYY-part3
- ZFS will then resilver the drive, to check if done run
intel_pstate
Trying to lower the idle power usage, I wanted to try a different cpu governor driver.
- check whether it is intel_pstate or intel_cpufreq
- apt install cpufrequtils
- cpufreq-info | head -n 13
- Determine if using grub or systemd-boot
- proxmox-boot-tool status
- efibootmgr -v
- to enable intel_pstate for grub boot:
- edit /etc/default/grub
- add intel_pstate=active to GRUB_CMDLINE_LINUX_DEFAULT
- example: GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=active"
- proxmox-boot-tool refresh
- wait for grub to finish updating and then reboot
- to enable intel_pstate for systemd-boot
- edit /etc/kernel/cmdline
- add intel_pstate=active to the end
- proxmox-boot-tool refresh
- wait for it to update and then reboot
- ensure it is now intel_pstate
- cpufreq-info | head -n 13
- unfortunately this did not change idle power usage
Fan Information
Here are the fan values from my r730xd
- 4% -> 2280 rpm -> 10 cfm
- 6% -> 2640 rpm -> 12 cfm
- 9% -> 3000 rpm -> 18 cfm
- 12% -> 3600 rpm -> 21 cfm
- 15% -> 4080 rpm -> 24 cfm
- 20% -> 4920 rpm -> 30 cfm
- 25% -> 5640 rpm -> 36 cfm
- 30% -> 6480 rpm -> 41 cfm
- 40% -> 7320 rpm -> 53 cfm
- 50% -> 8160 rpm -> 64 cfm
Appendix
Sources
- copying part tables
- intel_pstate:
