⚓ T7163 Convert Phacility file storage to use S3

Status	Assigned	Task
Resolved	epriestley	T7163 Convert Phacility file storage to use S3
Open	joshuaspence	T5155 Evaluate support for AWS IAM Roles in S3 Client
Resolved	epriestley	T7399 Fully separate live credentials from development repositories
Open	epriestley	T9828 Make migrating files less scary

epriestley created this task.Feb 4 2015, 6:23 PM

epriestley raised the priority of this task from to Needs Triage.

epriestley updated the task description. (Show Details)

epriestley moved this task to Backlog on the Phacility board.

epriestley added a subscriber: epriestley.

epriestley moved this task from Backlog to v1 Open Beta on the Phacility board.Feb 4 2015, 8:23 PM

epriestley added a revision: D11770: When cluster.instance is defined, use it to namespace S3 objects.Feb 15 2015, 12:55 AM

epriestley moved this task from v1 Open Beta to v0 Closed Beta on the Phacility board.Feb 15 2015, 1:02 AM

We should really use IAM Roles for this since putting root API keys on instances seems Very Questionable.

epriestley moved this task from v0 Closed Beta to v1 Open Beta on the Phacility board.Feb 16 2015, 6:44 PM

epriestley added a commit: rP3a8cd60bab41: When cluster.instance is defined, use it to namespace S3 objects.Feb 16 2015, 7:30 PM

btrahan triaged this task as Normal priority.Feb 17 2015, 10:51 PM

epriestley mentioned this in T7148: Allow users to export their data from Phacility.Feb 18 2015, 6:48 AM

epriestley moved this task from v1 Open Beta to Do After Launch on the Phacility board.Feb 20 2015, 8:39 PM

epriestley added a subtask: T7399: Fully separate live credentials from development repositories.Feb 27 2015, 3:43 PM

epriestley closed subtask T7399: Fully separate live credentials from development repositories as Resolved.Dec 14 2015, 6:39 PM

epriestley added a subtask: T9828: Make migrating files less scary.Dec 18 2015, 1:11 PM

Some of the older instances are beginning to collect enough data in MySQL to motivate this more strongly in the near term, especially as database data is currently stored multiple times on disk in backups.

epriestley moved this task from Backlog to vNext on the Files board.Jan 9 2016, 11:48 AM

We should be in good shape to move forward here after changes in T5155. My plan is roughly:

deploy the new stuff here for a bit;
then, start doing new writes to S3 in the cluster;
then, migrate older data off some of the bigger instances.

I'll probably do the actual migration stuff next weekend. We aren't facing any urgent pressure here, but are down to maybe ~2-ish months of headroom.

epriestley mentioned this in T10114: Upgrading: Amazon S3 Configuration Changes.Jan 10 2016, 4:44 PM

joshuaspence added a subscriber: joshuaspence.Jan 11 2016, 11:08 AM

This is now available in the cluster, but the storage engine currently never selects it for new writes because MySQL is still configured to allow 8MB chunks. I'm going to selectively migrate some data, but probably not turn default writes on for a week or so.

The second biggest source of storage pressure appears to be test instances that, e.g., imported the Linux Kernel and were then suspended. I've filed T10166 to follow up.