migrate · gcs → dosya
Move a bucket off Google Cloud Storage in four commands.
Two remotes, one copy, one check. rclone reads from Cloud Storage and writes to dosya.dev; nothing at the source is touched. When rclone check comes back clean, delete the bucket and stop the meter.
The run sheet
Fill in two names. Run it top to bottom.
Every line is safe to run twice. rclone skips whatever already matches, so a copy that stops can be started again with the same command. Nothing here deletes anything at GCS.
# a service account key with Storage Object Viewer on the bucket
$ rclone config create gcs "google cloud storage" \
service_account_file=/path/to/key.json# dosya is built into rclone: pick it in rclone config, or in one line
$ rclone config create dosya dosya \
api_key=dos_YOUR_API_KEY workspace_id=ws_abc123
# no dosya in your provider list? the S3 gateway works on any rclone version
# S3 keys are separate from the API key: generate them in dashboard settings
$ rclone config create dosya s3 provider=Other \
access_key_id=DOSYAXXXXXXXXXXXXXXXX secret_access_key=your_s3_secret_key \
endpoint=https://api.dosya.dev/s3 region=us-east-1 force_path_style=true
# native paths are dosya:<folder>
# gateway paths put the workspace first: dosya:ws_abc123/<folder>$ rclone copy gcs:my-bucket dosya:my-bucket --dry-run$ rclone copy gcs:my-bucket dosya:my-bucket \
--transfers 8 --checkers 16 --fast-list --progress
# prove it landed: size and hash compared on both sides
$ rclone check gcs:my-bucket dosya:my-bucketBefore you press enter
Two things about each end.
Read once, then run. Everything here shows up in the log if you skip it, so nothing is lost, only time.
At GCS
- Cold classes are slower to read. Nearline, Coldline and Archive objects are billed for retrieval and Archive ones take longer to come back. Check the bucket's storage class before you start.
- Requester Pays needs a project. On a Requester Pays bucket add --gcs-user-project YOUR_PROJECT to every command, or every read is refused.
At dosya.dev
- Empty files are refused. A 0-byte object is skipped with a warning and rclone check lists it, so you know which ones.
- Filenames stop at 255 characters. A longer key fails and shows up in the log; rename it at the source and run the copy again.
- Upload slots are per key. The Free plan allows 2 transfers in flight per API key, paid plans more. Above the cap rclone waits for a slot rather than failing, retrying every 5 seconds for up to 10 minutes.
Where GCS is the better fit
If your files feed BigQuery, Vertex or a GKE workload in the same region, keep them in Cloud Storage: in-region reads are free and the integrations are the point. Move the files people open in a browser.
Questions
Before you run it, answered.
- Is anything deleted in Cloud Storage?
- No. rclone copy only ever writes to dosya.dev; it never removes or changes anything at the source. Delete the bucket yourself once rclone check comes back clean.
- Can I use my own Google login instead of a service account?
- Yes. Run rclone config, pick Google Cloud Storage and leave service_account_file empty; rclone opens a browser to authorise. A service account is simpler on a server with no browser.
- Do I need a paid dosya.dev plan to migrate?
- No. The free plan has 5 GB and every protocol, rclone included. Copy one folder, run the check, and pick a plan when you know the size of what landed.
- What if the copy stops halfway?
- Run the same command again. rclone compares size and modification time, skips whatever already matches, and finishes the rest. rclone check at the end proves nothing was missed.
Copy one folder first.
5 GB free and every protocol on it. Run the sheet against one folder, read the check, then move the rest.