After migrating back to my Lenovo server, it didn't have enough drive bays for my backup drives. So I needed to use the Dell as the backup destination.
Two servers:
- Primary: this is the main server where the primary copy of the data exists (Lenovo)
- Secondary: this is the backup server where we will send the data to a read-only copy (Dell)
Setup SSH
On the primary server, create ssh key pair (if you already have a key pair feel free to use that):
- ssh-keygen -t ed25519 -C "root@primary"
- -C is just a comment to allow for easier identification, so feel free to change, but make sure you also change where it is used later
- cat id_ed25519.pub
- We will need this output for saving the key
On the secondary server, create our backup user and give it the public key:
- Install sudo
- apt install sudo -y
- Add the pveadmin user
- adduser pveadmin
- usermod -aG sudo pveadmin
- # Allow pveadmin to run certain commands without prompting for password so use visudo to add the line
- visudo
- pveadmin ALL=(ALL) NOPASSWD: /usr/sbin/zfs list *, /usr/sbin/zfs receive *, /usr/sbin/zpool scrub *
- Switch to the pveadmin user and add our ssh key
- su - pveadmin
- mkdir ~/.ssh
- chmod 700 ~/.ssh
- # Change the below to your key
- echo "ssh-ed25519 ABCD1234... root@primary" >> ~/.ssh/authorized_keys
- chmod 600 ~/.ssh/authorized_keys
- exit
- Test the ssh from primary to secondary and save the fingerprint
- ssh pveadmin@secondary
Send the backups
- Send over our initial data (use disown to prevent user disconnections from stopping job)
- zfs send -R --raw rpool/data@backup-20260323 | ssh pveadmin@secondary "sudo zfs receive -o readonly=on backup-pool/data" 2>errors.log & disown
- Send over incremental data
- zfs send -R --raw -I rpool/data@backup-20260323 rpool/data@backup-20260401 | ssh pveadmin@secondary "sudo zfs receive backup-pool/data" 2>errors.log & disown
Automate the backups
- Added a remote argument to my backup.sh script
Scripts
remote-backup.sh
#!/bin/sh
if [ $# -ne 3 ] ; then
echo "Requires 3 args source, destination, and user@host"
exit 1
fi
src=$1
dest=$2
remote=$3
snaplist=$(zfs list -t snapshot $src)
if [ $? -ne 0 ] ; then
echo "No snapshots found for $src"
exit 1
fi
bkpsnap=$(ssh ${remote} "sudo zfs list -t snapshot $dest 2>&1")
incremental=$?
if [ $incremental -eq 0 ] ; then
incsnap=$(echo "$bkpsnap" | tail -n 1 | cut -f1 -d" " | cut -f2 -d"@")
snap=$(echo "$snaplist" | grep "$incsnap" -A 1 | tail -n 1 | cut -f1 -d" " | cut -f2 -d"@")
if [ "$incsnap" = "$snap" ] ; then
echo "Backup is already up to date"
exit 1
fi
echo "Sending incremental snap $src@$snap on top of $dest@$incsnap"
echo "zfs send -R --raw -I $src@$incsnap $src@$snap | ssh ${remote} \"sudo zfs receive $dest\""
zfs send -R --raw -I $src@$incsnap $src@$snap | ssh ${remote} "sudo zfs receive $dest"
else
snap=$(echo "$snaplist" | tail -n +2 | head -n 1 | cut -f1 -d" " | cut -f2 -d"@")
echo "Sending initial snap $src@$snap to $dest"
echo "zfs send -R --raw $src@$snap | ssh ${remote} \"sudo zfs receive -o readonly=on $dest\""
zfs send -R --raw $src@$snap | ssh ${remote} "sudo zfs receive -o readonly=on $dest"
fi
echo "Complete\n"
Appendix
Sources
- https://www.ionos.com/digitalguide/server/configuration/how-to-enable-ssh-on-proxmox/
- https://github.com/alexandreravelli/Securing-SSH-Access-on-Proxmox-VE-9
- https://www.ibm.com/docs/en/b2b-integrator/6.2.0?topic=adapter-generate-new-ssh-user-identity-key
