Using rclone with S3-compatible storage

rclone is the tool most people end up using for scripted work against object storage. It syncs, mounts, copies between providers, and runs on everything.

Configure a remote

Run rclone config, choose n for a new remote, name it, and select the s3 backend. For the provider, pick Other for any S3-compatible service that is not one of the named ones.

You need three things: an access key ID, a secret access key, and the S3 endpoint URL.

The endpoint is what people get wrong. It must be the storage service’s S3 API endpoint, not the web address you log into in a browser. Those are usually different hosts, and pointing at the wrong one produces HTML error pages where XML was expected, which rclone reports as a confusing parse failure rather than a clean “wrong address”.

The resulting config:

[mystorage]
type = s3
provider = Other
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = https://s3.example.com

Test it immediately, before building anything on top:

rclone lsd mystorage:

If that lists your buckets, everything downstream will work. If it does not, fix it now - every later problem will be blamed on the wrong thing.

Copy and sync, and the difference that bites

copy adds and updates. It never deletes:

rclone copy ~/Documents mystorage:mybucket/documents --progress

sync makes the destination match the source, including deletions. It is the right command for backups and the wrong command to run casually:

rclone sync ~/Documents mystorage:mybucket/documents --dry-run

--dry-run prints what would happen and changes nothing. Running it before every sync costs seconds and prevents the one mistake in this tool that is genuinely painful - a source directory that failed to mount looks like an empty directory, and sync will faithfully empty the destination to match.

For real jobs, --max-delete 100 refuses to proceed if more than 100 files would be removed. It is the cheapest insurance available.

Flags worth knowing

--transfers N (default 4) - how many files move concurrently. Raising it to 16 or 32 helps enormously with many small files.

--checkers N (default 8) - how many comparisons run in parallel. Raise it when scanning large trees.

--fast-list - uses fewer, larger listing calls. On big buckets this is dramatically faster, at the cost of holding the listing in memory. Worth it above a few thousand objects.

--bwlimit 5M - caps bandwidth. Essential if the job shares a connection with people who need it. Accepts a schedule: --bwlimit "08:00,1M 18:00,off".

--exclude / --filter-from - keep junk out. .DS_Store, node_modules, *.tmp and build output are the usual suspects.

--progress - a live display. Omit it in cron jobs, where it produces unreadable logs.

--s3-chunk-size 64M - tunes multipart upload size for large files. The default is fine; raise it for very large objects on fast links.

Mounting a bucket

rclone can present a bucket as a filesystem:

rclone mount mystorage:mybucket ~/mnt/storage --vfs-cache-mode writes

--vfs-cache-mode writes is not optional in practice. Object storage cannot modify part of an object in place - objects are immutable and replaced whole. Any application that opens a file for random-access writing will fail without a local cache to work against. Leave it out and you get baffling errors from ordinary software.

This requires FUSE: macFUSE on macOS, WinFsp on Windows, built in on Linux. That install step is the reason mounting is not the default answer for non-technical colleagues - for them, native WebDAV on macOS or Windows needs no kernel extension at all.

Our protocol comparison covers the wider trade-off.

Verifying a transfer

rclone check compares source and destination:

rclone check ~/Documents mystorage:mybucket/documents --one-way

By default it compares sizes and hashes where both sides provide them. --one-way ignores extra files at the destination, which is what you want after a copy.

For anything you will later depend on, run this once. A backup you have not verified is a hypothesis.

Errors and what they actually mean

rclone’s errors are honest but terse. The common ones:

SignatureDoesNotMatch - the secret key is wrong, or a trailing space crept in when you pasted it. Occasionally a clock skew of more than 15 minutes on your machine, since SigV4 signatures are time-bound.

InvalidAccessKeyId - the key ID is wrong, or belongs to a different account than the endpoint you are pointing at.

NoSuchBucket - the bucket does not exist, or your credential cannot see it. These are indistinguishable by design: revealing which would leak information about buckets you have no access to.

A failure mentioning HTML or “expected XML” - you are pointing at a web address rather than the S3 API endpoint. This is the single most common misconfiguration.

RequestTimeTooSkewed - your system clock is wrong. Fix NTP.

Uploads succeed but the file is empty - almost always a shell redirection problem in the surrounding script rather than rclone.

Adding -vv to any command prints the full request and response, which resolves most of these in one run.

Scheduling it

For a real job, wrap the command in a script, log its output, and run it from cron or a systemd timer. The parts that make it dependable rather than merely working once - failure notification, restore testing, credential scoping - are covered in our guide to backing up a server to object storage.

One rclone-specific gotcha for cron: the config lives in your home directory, and cron’s HOME may not be what you expect. Pass it explicitly:

rclone sync /srv/data mystorage:backup --config /root/.config/rclone/rclone.conf