Mirroring a Drive folder to S3 every ten minutes

A shared Drive folder is the right place for people to put files. It is the wrong place to link to them from. A Drive link opens a preview page, asks for a login half the time, changes shape depending on who clicks it, and is useless as a src attribute or in a bulk upload. What I wanted was the folder as people use it, and a plain https://.../folder/file.mp4 for everything in it.

rclone does this in one command, and cron runs it every ten minutes.

The setup

Two rclone remotes: one for Drive, one for S3. The Drive one is set up with rclone config and an OAuth flow in the browser, once. The S3 one takes an access key with write rights on exactly one bucket, nothing else. Then the copy:

rclone sync drive:Creatives s3:my-public-bucket/creatives \
  --drive-shared-with-me=false \
  --s3-acl public-read \
  --transfers 4 --checkers 8 \
  --exclude "*.gdoc" --exclude "*.gsheet" --exclude "~$*" \
  --log-file ~/mirror.log --log-level NOTICE

Wrap it in flock so a slow run of a large video never overlaps with the next tick, and put it in crontab at */10 * * * *. That is the whole system.

sync versus copy

rclone sync makes the destination match the source, including deletions. If an editor removes a file from Drive, it disappears from S3 ten minutes later and every link to it breaks. rclone copy only adds and updates, never deletes. Which one you want depends on whether "it is gone from the folder" should mean "it is gone from the web". For a folder of ad creatives I wanted sync, because an old version lingering under a stable URL is exactly how the wrong file ends up in a campaign. For an archive I would use copy. Decide this before the first run, not after the first surprised message.

The things that bit me

Google-native documents are not files. A Google Doc in the folder has no bytes to copy; rclone can export it as .docx or skip it. I exclude them, because the folder is for finished media and a Doc in there is a mistake.

Spaces and umlauts in filenames become percent-encoded in the URL. The file is fine, the link is ugly and fragile when pasted. I asked the editors for a naming rule (lowercase, hyphens, no spaces) and added a tiny rename step before the sync for anything that breaks it. The rule stuck after a week.

Content types come from the extension. A .mp4 gets video/mp4 and plays inline; a file with no extension gets binary/octet-stream and downloads. Again, the naming rule.

Drive's API has a daily quota and rclone's default checker count is polite enough that a ten-minute cadence on a few thousand files never touches it. If the folder grows past that, --fast-list cuts the listing calls.

Public-read on a bucket is a decision. Nothing goes into that folder that should not be on the open internet, and the folder's name says so.

Why ten minutes

Fast enough that an editor can finish a file, wait for a coffee and paste the URL. Slow enough that a run almost never overlaps the next one. A webhook would be faster and would also be one more thing to break; a cron line is not.