We're using MySQL storage in Phacility right now; we should switch instances to use S3
Description
Revisions and Commits
Status | Assigned | Task | ||
---|---|---|---|---|
Resolved | epriestley | T7163 Convert Phacility file storage to use S3 | ||
Open | joshuaspence | T5155 Evaluate support for AWS IAM Roles in S3 Client | ||
Resolved | epriestley | T7399 Fully separate live credentials from development repositories | ||
Open | epriestley | T9828 Make migrating files less scary |
Event Timeline
We should really use IAM Roles for this since putting root API keys on instances seems Very Questionable.
Some of the older instances are beginning to collect enough data in MySQL to motivate this more strongly in the near term, especially as database data is currently stored multiple times on disk in backups.
We should be in good shape to move forward here after changes in T5155. My plan is roughly:
- deploy the new stuff here for a bit;
- then, start doing new writes to S3 in the cluster;
- then, migrate older data off some of the bigger instances.
I'll probably do the actual migration stuff next weekend. We aren't facing any urgent pressure here, but are down to maybe ~2-ish months of headroom.
This is now available in the cluster, but the storage engine currently never selects it for new writes because MySQL is still configured to allow 8MB chunks. I'm going to selectively migrate some data, but probably not turn default writes on for a week or so.
The second biggest source of storage pressure appears to be test instances that, e.g., imported the Linux Kernel and were then suspended. I've filed T10166 to follow up.