Page MenuHomePhabricator

Make storage backends extensible
Closed, ResolvedPublic

Description

For Haskell.org we're using Rackspace as our hosting provider, and it would be really swell if we could upload files to their Cloud Files (AKA OpenStack Swift) application, similarly to S3 support now.

We have a working implementation available here, but it requires a rather large dependency on php-opencloud for development (a concern from @epriestley) and is a fork. This (plus the small demand) makes upstreaming it more concerning, so it would be nice if storage backends could be loaded from libphutil libraries.

As far as I can tell this isn't currently possible since PhabricatorDefaultFileStorageEngineSelector currently has a hard-coded ordering for which backends to use (e.g. in our fork, Rackspace is last). So perhaps this wouldn't be too hard to fix in theory right now, but some other logic will need to be instituted to select storage engines.

Event Timeline

thoughtpolice raised the priority of this task from to Needs Triage.
thoughtpolice updated the task description. (Show Details)
thoughtpolice added a project: Phabricator.
thoughtpolice added a subscriber: thoughtpolice.

These are sort-of-extensible in a very effort-intensive way right now:

  • Subclass PhabricatorFileStorageEngine to implement your engine.
  • Subclass PhabricatorDefaultFileStorageEngineSelector to implement a new selector which selects your engine.
  • Put your engine and selector in a library.
  • Load it at runtime.
  • Set storage.engine-selector to choose your selector.

However, this is more complicated than it should be. This "selector class" pattern was one of the earliest approaches we used, and it wasn't that great. I think all other use cases were later replaced with the more modern extension patterns.

This one is a little tricky to replace because it has some unusual concerns: for example, it's reasonable to activate multiple engines to support rules like "very small files go in MySQL for quick access, larger stuff goes in S3", and we should ideally continue to access and read engines which we're no longer writing new files to. However, these use cases are not especially important, and they can still be accommodated with a more modern approach to storage engines.

Since this is technically possible today and there hasn't been much demand (and it doesn't immediately unblock anything else we want to do, and it looks like you have things working OK, albeit with a local patch) it probably won't happen for a while. It would be nice to get this cleaned up and it would make supporting installs that are changing storage a little easier, but it's hard to prioritize on those grounds alone.

That's fine with me. Ideally it would be nice to still fix this, and if we can get by with a hackish libphutil extension, I can live with that. However, I'm assigning this to myself since I may be able to hack something up eventually to support this better (it's been easier than I expected so far to work on Phab, anyway).

D12053 marks this as fixed. Specifically:

  • It removes the "selector" classes completely.
  • Engines must implement a few new methods so Phabricator can tell if they're writable, test engines, filesize limits, etc., and with what priority writes should be attempted to the engine.
  • Applicable engines are automatically discovered and selected at runtime and tried in priority order.

So the new requirements are:

  • Implement the new methods (all of which should be trivial).
  • Load the engine in a library somewhere.
  • It should show up on ApplicationsFilesHelp/Options, like this:

Screen_Shot_2015-03-12_at_12.28.29_PM.png (195×1 px, 37 KB)

  • If it's highlighted, that means it's recognized and considered writable.
  • You can delete your old Selector class, if you have one.