When i try to backup remote server with rsnapshot it errors out with 255 code

Every time I try running sudo rsnapshot -v alpha I get this type of error(it errors for every backup entry I have):

 ERROR: /usr/bin/rsync returned 255 while processing [email protected]:/etc/ /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \ --rsh=/usr/bin/ssh -i /home/user/ssh/id_ed25519 \ [email protected]:/usr/share/ \ /var/cache/rsnapshot/alpha.0/server_backup/ 

1. Yes, rsync installed on the server and this machine
2. Yes, rsync works if I try to manually copy some files with these root credentials from the remote
3. There is one thing that can be potentially it. When I tried to run the errored out command it demanded quotes around arguments for the rsh. Otherwise it would throw syntax error. But I am not sure how to forse rsnapshot to do that. And if I run the errored out command with quotes around the rsh key it will error with 255 code as well.
4. Firewall does not block ssh.
5. Server allows only public key authentication
6. I host my ubuntu server on Vultr

Here is my rsnapshot.conf file

 ################################################# # rsnapshot.conf - rsnapshot configuration file # ################################################# # # # PLEASE BE AWARE OF THE FOLLOWING RULE: # # # # This file requires tabs between elements # # # ################################################# ####################### # CONFIG FILE VERSION # ####################### config_version 1.2 ########################### # SNAPSHOT ROOT DIRECTORY # ########################### # All snapshots will be stored under this root directory. # snapshot_root /var/cache/rsnapshot/ # If no_create_root is enabled, rsnapshot will not automatically create the # snapshot_root directory. This is particularly useful if you are backing # up to removable media, such as a FireWire or USB drive. # #no_create_root 1 ################################# # EXTERNAL PROGRAM DEPENDENCIES # ################################# # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features. # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. # # See the README file or the man page for more details. # cmd_cp /bin/cp # uncomment this to use the rm program instead of the built-in perl routine. # cmd_rm /bin/rm # rsync must be enabled for anything to work. This is the only command that # must be enabled. # cmd_rsync /usr/bin/rsync # Uncomment this to enable remote ssh backups over rsync. # cmd_ssh /usr/bin/ssh # Comment this out to disable syslog support. # cmd_logger /usr/bin/logger # Uncomment this to specify the path to "du" for disk usage checks. # If you have an older version of "du", you may also want to check the # "du_args" parameter below. # #cmd_du /usr/bin/du # Uncomment this to specify the path to rsnapshot-diff. # #cmd_rsnapshot_diff /usr/bin/rsnapshot-diff # Specify the path to a script (and any optional arguments) to run right # before rsnapshot syncs files # #cmd_preexec /path/to/preexec/script # Specify the path to a script (and any optional arguments) to run right # after rsnapshot syncs files # #cmd_postexec /path/to/postexec/script # Paths to lvcreate, lvremove, mount and umount commands, for use with # Linux LVMs. # #linux_lvm_cmd_lvcreate /sbin/lvcreate #linux_lvm_cmd_lvremove /sbin/lvremove #linux_lvm_cmd_mount /bin/mount #linux_lvm_cmd_umount /bin/umount ######################################### # BACKUP LEVELS / INTERVALS # # Must be unique and in ascending order # # e.g. alpha, beta, gamma, etc. # ######################################### retain alpha 6 retain beta 7 retain gamma 4 #retain delta 3 ############################################ # GLOBAL OPTIONS # # All are optional, with sensible defaults # ############################################ # Verbose level, 1 through 5. # 1 Quiet Print fatal errors only # 2 Default Print errors and warnings only # 3 Verbose Show equivalent shell commands being executed # 4 Extra Verbose Show extra verbose information # 5 Debug mode Everything # verbose 2 # Same as "verbose" above, but controls the amount of data sent to the # logfile, if one is being used. The default is 3. # If you want the rsync output, you have to set it to 4 # loglevel 3 # If you enable this, data will be written to the file you specify. The # amount of data written is controlled by the "loglevel" parameter. # logfile /var/log/rsnapshot.log # If enabled, rsnapshot will write a lockfile to prevent two instances # from running simultaneously (and messing up the snapshot_root). # If you enable this, make sure the lockfile directory is not world # writable. Otherwise anyone can prevent the program from running. # lockfile /var/run/rsnapshot.pid # By default, rsnapshot check lockfile, check if PID is running # and if not, consider lockfile as stale, then start # Enabling this stop rsnapshot if PID in lockfile is not running # #stop_on_stale_lockfile 0 # Default rsync args. All rsync commands have at least these options set. # #rsync_short_args -a #rsync_long_args --delete --numeric-ids --relative --delete-excluded # ssh has no args passed by default, but you can specify some here. # ssh_args -i /home/user/ssh/id_ed25519 # Default arguments for the "du" program (for disk space reporting). # The GNU version of "du" is preferred. See the man page for more details. # If your version of "du" doesn't support the -h flag, try -k flag instead. # #du_args -csh # If this is enabled, rsync won't span filesystem partitions within a # backup point. This essentially passes the -x option to rsync. # The default is 0 (off). # #one_fs 0 # The include and exclude parameters, if enabled, simply get passed directly # to rsync. If you have multiple include/exclude patterns, put each one on a # separate line. Please look up the --include and --exclude options in the # rsync man page for more details on how to specify file name patterns. # #include ??? #include ??? #exclude ??? #exclude ??? # The include_file and exclude_file parameters, if enabled, simply get # passed directly to rsync. Please look up the --include-from and # --exclude-from options in the rsync man page for more details. # #include_file /path/to/include/file #exclude_file /path/to/exclude/file # If your version of rsync supports --link-dest, consider enabling this. # This is the best way to support special files (FIFOs, etc) cross-platform. # The default is 0 (off). # #link_dest 0 # When sync_first is enabled, it changes the default behaviour of rsnapshot. # Normally, when rsnapshot is called with its lowest interval # (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest # intervals. With sync_first enabled, "rsnapshot sync" handles the file sync, # and all interval calls simply rotate files. See the man page for more # details. The default is 0 (off). # #sync_first 0 # If enabled, rsnapshot will move the oldest directory for each interval # to [interval_name].delete, then it will remove the lockfile and delete # that directory just before it exits. The default is 0 (off). # #use_lazy_deletes 0 # Number of rsync re-tries. If you experience any network problems or # network card issues that tend to cause ssh to fail with errors like # "Corrupted MAC on input", for example, set this to a non-zero value # to have the rsync operation re-tried. # #rsync_numtries 0 # LVM parameters. Used to backup with creating lvm snapshot before backup # and removing it after. This should ensure consistency of data in some special # cases # # LVM snapshot(s) size (lvcreate --size option). # #linux_lvm_snapshotsize 100M # Name to be used when creating the LVM logical volume snapshot(s). # #linux_lvm_snapshotname rsnapshot # Path to the LVM Volume Groups. # #linux_lvm_vgpath /dev # Mount point to use to temporarily mount the snapshot(s). # #linux_lvm_mountpath /path/to/mount/lvm/snapshot/during/backup ############################### ### BACKUP POINTS / SCRIPTS ### ############################### # REMOTE SERVER backup [email protected]:/home/ server_backup/ backup [email protected]:/etc/ server_backup/ #backup_script /usr/local/bin/backup_pgsql.sh localhost/postgres/ # You must set linux_lvm_* parameters below before using lvm snapshots #backup lvm://vg0/xen-home/ lvm-vg0/xen-home/ # EXAMPLE.COM #backup_exec /bin/date "+ backup of example.com started at %c" #backup [email protected]:/home/ example.com/ +rsync_long_args=--bwlimit=16,exclude=core #backup [email protected]:/etc/ example.com/ exclude=mtab,exclude=core #backup_exec ssh [email protected] "mysqldump -A > /var/db/dump/mysql.sql" #backup [email protected]:/var/db/dump/ example.com/ #backup_exec /bin/date "+ backup of example.com ended at %c" # CVS.SOURCEFORGE.NET #backup_script /usr/local/bin/backup_rsnapshot_cvsroot.sh rsnapshot.cvs.sourceforge.net/ # RSYNC.SAMBA.ORG #backup rsync://rsync.samba.org/r syncftp/ rsync.samba.org/rsyncftp/ 

My sshd logs look this way:

2025-01-21T16:47:06.445342+00:00 server sshd[2069]: Connection from 99.11.11.11 port 57908 on 151.131.222.222 port 22 rdomain "" 2025-01-21T16:47:06.445890+00:00 server sshd[2069]: debug1: Local version string SSH-2.0-OpenSSH_9.7p1 Ubuntu-7ubuntu4 2025-01-21T16:47:06.446150+00:00 server sshd[2069]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.10 2025-01-21T16:47:06.446387+00:00 server sshd[2069]: debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.10 pat OpenSSH* compat 0x04000000 2025-01-21T16:47:06.448025+00:00 server sshd[2069]: debug1: permanently_set_uid: 109/65534 [preauth] 2025-01-21T16:47:06.448401+00:00 server sshd[2069]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] 2025-01-21T16:47:06.448865+00:00 server sshd[2069]: debug1: SSH2_MSG_KEXINIT sent [preauth] 2025-01-21T16:47:06.473088+00:00 server sshd[2069]: debug1: SSH2_MSG_KEXINIT received [preauth] 2025-01-21T16:47:06.473305+00:00 server sshd[2069]: debug1: kex: algorithm: curve25519-sha256 [preauth] 2025-01-21T16:47:06.473602+00:00 server sshd[2069]: debug1: kex: host key algorithm: ssh-ed25519 [preauth] 2025-01-21T16:47:06.473829+00:00 server sshd[2069]: debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [preauth] 2025-01-21T16:47:06.474193+00:00 server sshd[2069]: debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none [preauth] 2025-01-21T16:47:06.474496+00:00 server sshd[2069]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth] 2025-01-21T16:47:06.502026+00:00 server sshd[2069]: debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth] 2025-01-21T16:47:06.509345+00:00 server sshd[2069]: debug1: ssh_packet_send2_wrapped: resetting send seqnr 3 [preauth] 2025-01-21T16:47:06.509768+00:00 server sshd[2069]: debug1: rekey out after 134217728 blocks [preauth] 2025-01-21T16:47:06.510085+00:00 server sshd[2069]: debug1: SSH2_MSG_NEWKEYS sent [preauth] 2025-01-21T16:47:06.510210+00:00 server sshd[2069]: debug1: Sending SSH2_MSG_EXT_INFO [preauth] 2025-01-21T16:47:06.510573+00:00 server sshd[2069]: debug1: expecting SSH2_MSG_NEWKEYS [preauth] 2025-01-21T16:47:06.543286+00:00 server sshd[2069]: debug1: ssh_packet_read_poll2: resetting read seqnr 3 [preauth] 2025-01-21T16:47:06.543606+00:00 server sshd[2069]: debug1: SSH2_MSG_NEWKEYS received [preauth] 2025-01-21T16:47:06.543946+00:00 server sshd[2069]: debug1: rekey in after 134217728 blocks [preauth] 2025-01-21T16:47:06.544260+00:00 server sshd[2069]: debug1: KEX done [preauth] 2025-01-21T16:47:06.636933+00:00 server sshd[2069]: debug1: userauth-request for user root service ssh-connection method none [preauth] 2025-01-21T16:47:06.637064+00:00 server sshd[2069]: debug1: attempt 0 failures 0 [preauth] 2025-01-21T16:47:06.638069+00:00 server sshd[2069]: debug1: PAM: initializing for "root" 2025-01-21T16:47:06.641531+00:00 server sshd[2069]: debug1: PAM: setting PAM_RHOST to "99.11.11.11" 2025-01-21T16:47:06.642045+00:00 server sshd[2069]: debug1: PAM: setting PAM_TTY to "ssh" 2025-01-21T16:47:06.664190+00:00 server sshd[2069]: Connection closed by authenticating user root 99.11.11.11 port 57908 [preauth] 2025-01-21T16:47:06.665162+00:00 server sshd[2069]: debug1: do_cleanup [preauth] 2025-01-21T16:47:06.666011+00:00 server sshd[2069]: debug1: monitor_read_log: child log fd closed 2025-01-21T16:47:06.666354+00:00 server sshd[2069]: debug1: do_cleanup 2025-01-21T16:47:06.666609+00:00 server sshd[2069]: debug1: PAM: cleanup 2025-01-21T16:47:06.667644+00:00 server sshd[2069]: debug1: Killing privsep child 2070 2025-01-21T16:47:06.668031+00:00 server sshd[2069]: debug1: audit_event: unhandled event 12 

My iptables rules look like this:

Chain INPUT (policy DROP) target prot opt source destination ufw-before-logging-input all -- anywhere anywhere ufw-before-input all -- anywhere anywhere ufw-after-input all -- anywhere anywhere ufw-after-logging-input all -- anywhere anywhere ufw-reject-input all -- anywhere anywhere ufw-track-input all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination ufw-before-logging-forward all -- anywhere anywhere ufw-before-forward all -- anywhere anywhere ufw-after-forward all -- anywhere anywhere ufw-after-logging-forward all -- anywhere anywhere ufw-reject-forward all -- anywhere anywhere ufw-track-forward all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination ufw-before-logging-output all -- anywhere anywhere ufw-before-output all -- anywhere anywhere ufw-after-output all -- anywhere anywhere ufw-after-logging-output all -- anywhere anywhere ufw-reject-output all -- anywhere anywhere ufw-track-output all -- anywhere anywhere Chain ufw-after-forward (1 references) target prot opt source destination Chain ufw-after-input (1 references) target prot opt source destination ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:netbios-ns ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:netbios-dgm ufw-skip-to-policy-input tcp -- anywhere anywhere tcp dpt:netbios-ssn ufw-skip-to-policy-input tcp -- anywhere anywhere tcp dpt:microsoft-ds ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:bootps ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:bootpc ufw-skip-to-policy-input all -- anywhere anywhere ADDRTYPE match dst-type BROADCAST Chain ufw-after-logging-forward (1 references) target prot opt source destination Chain ufw-after-logging-input (1 references) target prot opt source destination LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warn prefix "[UFW BLOCK] " Chain ufw-after-logging-output (1 references) target prot opt source destination Chain ufw-after-output (1 references) target prot opt source destination Chain ufw-before-forward (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere icmp destination-unreachable ACCEPT icmp -- anywhere anywhere icmp time-exceeded ACCEPT icmp -- anywhere anywhere icmp parameter-problem ACCEPT icmp -- anywhere anywhere icmp echo-request ufw-user-forward all -- anywhere anywhere Chain ufw-before-input (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ufw-logging-deny all -- anywhere anywhere ctstate INVALID DROP all -- anywhere anywhere ctstate INVALID ACCEPT icmp -- anywhere anywhere icmp destination-unreachable ACCEPT icmp -- anywhere anywhere icmp time-exceeded ACCEPT icmp -- anywhere anywhere icmp parameter-problem ACCEPT icmp -- anywhere anywhere icmp echo-request ACCEPT udp -- anywhere anywhere udp spt:bootps dpt:bootpc ufw-not-local all -- anywhere anywhere ACCEPT udp -- anywhere mdns.mcast.net udp dpt:mdns ACCEPT udp -- anywhere 239.200.200.200 udp dpt:1900 ufw-user-input all -- anywhere anywhere Chain ufw-before-logging-forward (1 references) target prot opt source destination Chain ufw-before-logging-input (1 references) target prot opt source destination Chain ufw-before-logging-output (1 references) target prot opt source destination Chain ufw-before-output (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ufw-user-output all -- anywhere anywhere Chain ufw-logging-allow (0 references) target prot opt source destination LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warn prefix "[UFW ALLOW] " Chain ufw-logging-deny (2 references) target prot opt source destination RETURN all -- anywhere anywhere ctstate INVALID limit: avg 3/min burst 10 LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warn prefix "[UFW BLOCK] " Chain ufw-not-local (1 references) target prot opt source destination RETURN all -- anywhere anywhere ADDRTYPE match dst-type LOCAL RETURN all -- anywhere anywhere ADDRTYPE match dst-type MULTICAST RETURN all -- anywhere anywhere ADDRTYPE match dst-type BROADCAST ufw-logging-deny all -- anywhere anywhere limit: avg 3/min burst 10 DROP all -- anywhere anywhere Chain ufw-reject-forward (1 references) target prot opt source destination Chain ufw-reject-input (1 references) target prot opt source destination Chain ufw-reject-output (1 references) target prot opt source destination Chain ufw-skip-to-policy-forward (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain ufw-skip-to-policy-input (7 references) target prot opt source destination DROP all -- anywhere anywhere Chain ufw-skip-to-policy-output (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain ufw-track-forward (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere ctstate NEW ACCEPT udp -- anywhere anywhere ctstate NEW Chain ufw-track-input (1 references) target prot opt source destination Chain ufw-track-output (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere ctstate NEW ACCEPT udp -- anywhere anywhere ctstate NEW Chain ufw-user-forward (1 references) target prot opt source destination Chain ufw-user-input (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ACCEPT udp -- anywhere anywhere udp dpt:openvpn ACCEPT tcp -- anywhere anywhere tcp dpt:ssh /* 'dapp_OpenSSH' */ ACCEPT tcp -- anywhere anywhere tcp dpt:https ACCEPT tcp -- anywhere anywhere tcp dpt:http ACCEPT udp -- anywhere dns.google udp dpt:domain ACCEPT tcp -- anywhere dns.google tcp dpt:domain Chain ufw-user-limit (0 references) target prot opt source destination LOG all -- anywhere anywhere limit: avg 3/min burst 5 LOG level warn prefix "[UFW LIMIT BLOCK] " REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain ufw-user-limit-accept (0 references) target prot opt source destination ACCEPT all -- anywhere anywhere Chain ufw-user-logging-forward (0 references) target prot opt source destination Chain ufw-user-logging-input (0 references) target prot opt source destination Chain ufw-user-logging-output (0 references) target prot opt source destination Chain ufw-user-output (1 references) target prot opt source destination 

My sshd_config file:

PermitRootLogin yes # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Include /etc/ssh/sshd_config.d/*.conf # When systemd socket activation is used (the default), the socket # configuration must be re-generated after changing Port, AddressFamily, or # ListenAddress. # # For changes to take effect, run: # # systemctl daemon-reload # systemctl restart ssh.socket # #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key #HostKey /etc/ssh/ssh_host_ed25519_key # Ciphers and keying #RekeyLimit default none # Logging #SyslogFacility AUTH LogLevel DEBUG # Authentication: #LoginGraceTime 2m #PermitRootLogin prohibit-password #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 PubkeyAuthentication yes # Expect .ssh/authorized_keys2 to be disregarded by default in future. #AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 #AuthorizedPrincipalsFile none #AuthorizedKeysCommand none #AuthorizedKeysCommandUser nobody # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # To disable tunneled clear text passwords, change to no here! PasswordAuthentication no #PermitEmptyPasswords no # Change to yes to enable challenge-response passwords (beware issues with # some PAM modules and threads) KbdInteractiveAuthentication no # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes #GSSAPIStrictAcceptorCheck yes #GSSAPIKeyExchange no # Set this to 'yes' to enable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the KbdInteractiveAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via KbdInteractiveAuthentication may bypass # the setting of "PermitRootLogin prohibit-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and KbdInteractiveAuthentication to 'no'. UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PermitTTY yes PrintMotd no #PrintLastLog yes #TCPKeepAlive yes #PermitUserEnvironment no #Compression delayed #ClientAliveInterval 0 #ClientAliveCountMax 3 #UseDNS no #PidFile /run/sshd.pid #MaxStartups 10:30:100 #PermitTunnel no #ChrootDirectory none #VersionAddendum none # no default banner path #Banner none # Allow client to pass locale environment variables AcceptEnv LANG LC_* # override default of no subsystems Subsystem sftp /usr/lib/openssh/sftp-server # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # PermitTTY no # ForceCommand cvs server 

Thank you for your attention, hopefully someone can help me solve this. I spent the whole day on this issue.

If you think I can use another tool for the backups, please let me know. I am new to system administrating so will gladly appreciate any help!