Replacing drives in ZFS take two

As it had been ~3 years since the last storage upgrade and there was only about 20% free space remaining, I wanted to take advantage of Black Friday prices. From WD.com, I grabbed 2 x 16TB Red Pro drives with a 5 year warranty.

Verify the drives

• smartctl short tests
• smartctl -t short /dev/sdX
• smartctl -t short /dev/sdY
• check smartctl test status after ~5 minutes
• smartctl -a /dev/sdX
• smartctl -a /dev/sdY
• badblocks
• Does 4 iterations of a write followed by a read. For my 16TB drives, each step took ~24 hours for a total of around 8 days.
• nohup badblocks -wsv -b 4096 /dev/sdX >16TB_serial_1.log 2>&1 &
• nohup badblocks -wsv -b 4096 /dev/sdY >16TB_serial_2.log 2>&1 &
• The logs will continue to grow as it runs. Mine got to 5.3MiB each for my 16TB drives after ~8 days
• Create script to check status of verification
• echo "" > newline.log
• echo "cat 16TB_serial_1.log newline.log 16TB_serial_2.log newline.log" > status.sh
• sh status.sh

Add the drives to the mirror

• Attach the drives
• zpool attach [POOLNAME] [EXISTING_DRIVE] [NEW_DRIVE_1]
• zpool attach [POOLNAME] [EXISTING_DRIVE] [NEW_DRIVE_2]
• Resilver both drives at once
• resolves: (awaiting resilver) on [NEW_DRIVE_2]
• zpool resilver [POOLNAME]
• Check on resilver
• zpool status [POOLNAME]
• resilver in progress since Wed Dec 18 17:37:48 2024
• I had 4.35TiB to resilver with an estimated runtime of around 7 hours, it ~7 hours and 45 minutes

Remove the old drives

I am planning on waiting at least 11 days to ensure the new drives are functioning as expected, but this is the step to remove them.

• zpool detach [POOLNAME] [DISKNAME]

Appendix

Sources

• https://wiki.archlinux.org/title/Badblocks
• https://www.reddit.com/r/zfs/comments/10zu5r6/zfs_and_resilvering_multiple_drives_at_once/