Sending ZFS backups to another server

After migrating back to my Lenovo server, it didn't have enough drive bays for my backup drives. So I needed to use the Dell as the backup destination.

Two servers:

• Primary: this is the main server where the primary copy of the data exists (Lenovo)
• Secondary: this is the backup server where we will send the data to a read-only copy (Dell)

Setup SSH

On the primary server, create ssh key pair (if you already have a key pair feel free to use that):

• ssh-keygen -t ed25519 -C "root@primary"
• -C is just a comment to allow for easier identification, so feel free to change, but make sure you also change where it is used later
• cat id_ed25519.pub
• We will need this output for saving the key

On the secondary server, create our backup user and give it the public key:

• Install sudo
• apt install sudo -y
• Add the pveadmin user
• adduser pveadmin
• usermod -aG sudo pveadmin
• # Allow pveadmin to run certain commands without prompting for password so use visudo to add the line
• visudo
• pveadmin ALL=(ALL) NOPASSWD: /usr/sbin/zfs list *, /usr/sbin/zfs receive *, /usr/sbin/zpool scrub *
• Switch to the pveadmin user and add our ssh key
• su - pveadmin
• mkdir ~/.ssh
• chmod 700 ~/.ssh
• # Change the below to your key
• echo "ssh-ed25519 ABCD1234... root@primary" >> ~/.ssh/authorized_keys
• chmod 600 ~/.ssh/authorized_keys
• exit
• Test the ssh from primary to secondary and save the fingerprint
• ssh pveadmin@secondary

Send the backups

• Send over our initial data (use disown to prevent user disconnections from stopping job)
• zfs send -R --raw rpool/data@backup-20260323 | ssh pveadmin@secondary "sudo zfs receive -o readonly=on backup-pool/data" 2>errors.log & disown
• Send over incremental data
• zfs send -R --raw -I rpool/data@backup-20260323 rpool/data@backup-20260401 | ssh pveadmin@secondary "sudo zfs receive backup-pool/data" 2>errors.log & disown

Automate the backups

• Added a remote argument to my backup.sh script

Scripts

remote-backup.sh

#!/bin/sh

if [ $# -ne 3 ] ; then

  echo "Requires 3 args source, destination, and user@host"

  exit 1

fi

src=$1

dest=$2

remote=$3

snaplist=$(zfs list -t snapshot $src)

if [ $? -ne 0 ] ; then

  echo "No snapshots found for $src"

  exit 1

fi

bkpsnap=$(ssh ${remote} "sudo zfs list -t snapshot $dest 2>&1")

incremental=$?

if [ $incremental -eq 0 ] ; then

  incsnap=$(echo "$bkpsnap" | tail -n 1 | cut -f1 -d" " | cut -f2 -d"@")

  snap=$(echo "$snaplist" | grep "$incsnap" -A 1 | tail -n 1 | cut -f1 -d" " | cut -f2 -d"@")

  if [ "$incsnap" = "$snap" ] ; then

    echo "Backup is already up to date"

    exit 1

  fi

  echo "Sending incremental snap $src@$snap on top of $dest@$incsnap"

  echo "zfs send -R --raw -I $src@$incsnap $src@$snap | ssh ${remote} \"sudo zfs receive $dest\""

  zfs send -R --raw -I $src@$incsnap $src@$snap | ssh ${remote} "sudo zfs receive $dest"

else

  snap=$(echo "$snaplist" | tail -n +2 | head -n 1 | cut -f1 -d" " | cut -f2 -d"@")

  echo "Sending initial snap $src@$snap to $dest"

  echo "zfs send -R --raw $src@$snap | ssh ${remote} \"sudo zfs receive -o readonly=on $dest\""

  zfs send -R --raw $src@$snap | ssh ${remote} "sudo zfs receive -o readonly=on $dest"

fi

echo "Complete\n"

Appendix

Sources

• https://www.ionos.com/digitalguide/server/configuration/how-to-enable-ssh-on-proxmox/
• https://github.com/alexandreravelli/Securing-SSH-Access-on-Proxmox-VE-9
• https://www.ibm.com/docs/en/b2b-integrator/6.2.0?topic=adapter-generate-new-ssh-user-identity-key