0

I am using rsync to sync around ~2TB of data to a mechanical HDD-5400RPM. It's taking way too long.

How can I improve the sync and check speeds to get as close as possible to maximum/optimal?

Here is the rsync sync command I am running with the flags:

rsync -vvv \ # -v: Basic information, like the files being transferred. # -vv: More details, such as when files are being skipped. # -vvv: Even more detailed, including internal operations and potentially overwhelming. -z \ # Compress data during transfer -h \ # Human-readable -a \ # Archive mode -A \ # Preserve ACLs -H \ # Preserve hard links -S \ # Handle sparse files efficiently -X \ # Preserve extended attributes "$SRC" \ "$DEST_DIR" \ --log-file=$log_file \ --progress \ --links \ --checksum \ --bwlimit="192M" \ --delete-during \ --delete \ --timeout="60" \ --compress \ --no-o \ --no-g \ --whole-file \ --preallocate \ --atimes \ --update \ --partial

And here is the rsync check command I am running with the flags:

rsync -vvv \ # -v: Basic information, like the files being transferred. # -vv: More details, such as when files are being skipped. # -vvv: Even more detailed, including internal operations and potentially overwhelming. -z \ # Compress data during transfer -h \ # Human-readable -a \ # Archive mode -A \ # Preserve ACLs -H \ # Preserve hard links -S \ # Handle sparse files efficiently -X \ # Preserve extended attributes "$src" \ "$dest" \ --log-file=$log_file --progress \ --links \ --checksum \ --bwlimit="192M" \ --compress --no-o \ --no-g \ --contimeout="60" \ --ignore-errors \ --stats \ --max-delete=0 \ --whole-file \ --preallocate \ --atimes \ --update

Rsync version:

rsync --version rsync version 3.2.7 protocol version 31 Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, symlinks, symtimes, hardlinks, hardlink-specials, hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs, xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes Optimizations: SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5 Checksum list: xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none Compress list: zstd lz4 zlibx zlib none Daemon auth list: sha512 sha256 sha1 md5 md4 rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.

Ubuntu Server version:

ubuntu 24.04 (64 bit) LTS
Improve this question
16
  • 2
    Don't compress when the media between the read and write is not slow. It sounds like you're using rsync to copy from one disk to another on the same server, so the "media" in between would be a simple pipeline and not slow. Some of your slowness may be coming from compression causing a cpu bottleneck, so removing the compress flags may give you an increase in throughput. (I think it will give you better throughput, but I can't say how much. Maybe a lot, maybe only a little) Commented Aug 19, 2024 at 15:42
  • 1
    Is this over a network or local? Also I notice - other than 5400rpm HDDs being pretty slow - you have a bandwidth limit set up Commented Aug 19, 2024 at 15:44
  • 1
    @atkuzmanov yes. The rsync process is reading $SRC as a filesystem, and rsync's --compress flag does not affect the data read from filesystems (or written to them), it only affects the data transmitted from the sending rsync process to the receiving rsync process. Commented Aug 19, 2024 at 15:53
  • 1
    @atkuzmanov could you add a speed estimate for writing to your HDD to your question? If in doubt, I usually test using yes | pv -S -s "32G" > /target/tmpfilename . 32G here is twice the RAM size of the machine to make sure we're not just writing to in-RAM write buffers. Commented Aug 19, 2024 at 15:58
  • 1
    Notice that modern cpus can saturate a gigabit connection over ssh because of hardware support. Not so with gzip invoked with -z Commented Aug 19, 2024 at 16:12

1 Answer 1

Reset to default
2

If this is between local drives? I'd almost suggest using cp, then using rsync to ensure there's no errors.

Other than that, breaking it down

You don't need all these flags

I'm not going to go through each one and tell you why they're a bad idea, but there's a few that stand out

--bwlimit="192M" \ Sets a bandwidth limit of 192M - considering even a 'old' slow hard drive can beat this handily, this is probably the first thing that's hamstringing you.

If its a SMB mount, its over the network, compression can make sense to save bandwidth, and potentially improve speed but not everything is equally compressible, and on modern machines, its probably worth weighing/testing it out with and without it.

--whole-file, -W copy files whole (w/o delta-xfer algorithm)

This means you're transferring the whole file, not just what's changed

Then you're running

--partial keep partially transferred files

This makes no sense since you're downloading the whole lot each time anyway. I'd remove both - let rsync do its magic if a transfer fails or a file changes. Else you might as well just cp , If you're using rsync to keep things synced or to pick up on a failed transfer, whole file makes no sense. Delta transfers are awesome

I also suspect running only one of --delete or --delete-during would be useful - as opposed to both.

I also notice you're telling it to preserve attributes -a which includes preserving the owner and group, then running -no-o and -no-g. Not sure which'll win out but since -a is rlptgoD, picking whichever flags you need makes sense over picking -a then removing some attribute sync options.

At this point, after pointing out the most obvious things - I'd suggest paring down your rsync command to a minimum - referring to the man page, and adding only what you need, based off the answer or the commands.

Improve this answer
7
  • 1
    nope, the compression of an SMB mount doesn't happen over the network; essentially, for the rsync process, the source file is local. It gets requested from the server via SMB by the kernel, not via the rsync protocol by rsync. So, compression here never helps. Commented Aug 19, 2024 at 16:08
  • 2
    Compression is only relevant if bandwidth between the two rsync instances is much less than disk speed. Commented Aug 19, 2024 at 16:13
  • Thank you very much for your detailed reply! The source system is a locally mounted TrueNas Scale SMB share. And the destination file system is a local ExFAT HDD attached via SATA via a SAS Controller. It did not see any speeds coming close to the bwlimit, I currently set it to try and throttle an SSD which is also attached as a destination for some files, but I will remove it. --whole-file - I have a mixture of lots of small and some big files, thought I should add both flags for both types of files... Commented Aug 20, 2024 at 14:03
  • 1
    @atkuzmanov all these options do more harm than good. How about you stop trying to prematurely optimize? In your other question you figured out that exFAT is not a viable file system for what you're doing, so use a different file system, use the absolute standard options for rsync, and then compare. Obviously, the people that wrote rsync are not totally clueless about how to make rsync run performant, so that default options are usually good. Commented Aug 20, 2024 at 15:04
  • 1
    Usually -a suffices for copying Commented Aug 20, 2024 at 15:39

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.