Migrating Arq to Google Coldline from Nearline

I’ve been a long time user of Arq Backup; since they introduced their Amazon Glacier storage. I later migrated to Google Nearline once available due to the lower latency, and now I want to use coldline for the 30% cost savings. Google doesn’t necessarily make it easy, but after some scouring of the internet, I’ve come up with a relatively straightfoward gist to migrate from nearline to coldline. Some assembly required.

I originally was using the gsutil rewrite command¹ but this really wasn’t effective at the scale I needed so I went with the lifecycle setting, which takes ~24 hours to take effect.

You will also need to change your default storage object for all new objects.

UPDATE 1/7/2017 using age doesn’t appear to cause the lifecycle to kick in, so createdBefore is the way to go.

To diagnose you can use logging + bigquery to make sure the conversion is taking place. you can follow the instructions from google and use the sql query + shell script below.

1. Stack Overflow reference to changing storage class of existing data ↩