MetaMTA - lay some ground work for having an application
ClosedPublic
Actions

Authored by btrahan on Jun 22 2015, 8:21 PM.

Details

Reviewers

epriestley

Maniphest Tasks

T5791: Write Herald rules for outbound mail

Commits

Restricted Diffusion Commit
rPea5462fb600b: MetaMTA - lay some ground work for having an application

Summary

Ref T5791. This does a few bits there. Namely:

Adds PHID column to PhabricatorMetaMTAMail
- Implements a PhabricatorMetaMTAMailPHIDType
- Script to backpopulate them.
Makes PhabricatorMetaMTAMail implement PolicyInterface.
- View policy is NOONE and the author and recipients have automatic view capabilities
- No edit capability.
Adds a PhabricatorMetaMTAMailQuery for PhabricatorMetaMTAMail.

Test Plan

ran ./bin/storage upgrade successfully. commented on a maniphest task and verifed the metamta mail object in the database was created successfully with a shiny new phid

Diff Detail

Repository

rP Phabricator

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

btrahan updated this revision to Diff 32429.Jun 22 2015, 8:21 PM

btrahan added a task: T5791: Write Herald rules for outbound mail.

btrahan retitled this revision from to MetaMTA - lay some ground work for having an application.

btrahan updated this object.

btrahan edited the test plan for this revision. (Show Details)

btrahan added a reviewer: epriestley.

Herald added subscribers: Korvin, epriestley. · View Herald TranscriptJun 22 2015, 8:21 PM

epriestley accepted this revision.Jun 22 2015, 8:27 PM

epriestley edited edge metadata.

epriestley added inline comments.

src/applications/metamta/phid/PhabricatorMetaMTAMailPHIDType.php
41	We may want to hold `$subject` out of this for now, until stuff is a little more mature, although I don't think they're ever actually sensitive.
src/applications/metamta/query/PhabricatorMetaMTAMailQuery.php
20	You may be able to get away with implementing `newResultObject()`, then just having this do: return $this->loadStandardPage($this->newResultObject());
62–64	I think you don't need this if you implement `getPrimaryTableAlias()`.
src/applications/metamta/storage/PhabricatorMetaMTAMail.php
1018	You may need to call `expandRecipients()` on this, too -- I think `getAllActorPHIDs()` includes recipients like projects in raw form, as `PHID-PROJ-xyz`. `expandRecipients()` will expand them into the underlying users, `PHID-USER-xyz`. This should be moot once edges show up.

This revision is now accepted and ready to land.Jun 22 2015, 8:27 PM

updates - thanks!

Closed by commit rPea5462fb600b: MetaMTA - lay some ground work for having an application (authored by btrahan, committed by btrahan). · Explain WhyJun 22 2015, 8:45 PM

This revision was automatically updated to reflect the committed changes.

Is it expected that these migrations will take a long time (like > 30 minutes)?

Yeah, I added some guidance on that here:

https://secure.phabricator.com/w/changelog/2015.26/

This period includes mail migrations which may take some time to complete on larger installs.

Not very helpful, but we don't really have much of a way to guess how long things will take because there's huge variance (like, 100x) between installs depending on configuration, hardware, size of the data, etc.

(But I'd be surprised if they took too much longer than ~30 minutes, even on large installs -- I think they took ~8 minutes here, and I recall a couple 15-20 minute reports on fairly active installs.)

One relatively easy thing I think we could do which would often improve the state of the world here is to have MigrationIterator render a progress bar automatically or with some flag. That's less useful diagnostically than emitting output, but in practice the breakdown of issues with migrations is ~100% installs not being able to predict how long they'll take and ~0% them breaking halfway through.

From memory I'm pretty sure this was a migration that took so long for us that I had to cancel it and truncate the mail table so it wouldn't have to migrate data.

For posterity:

For rough ballparks, a gzipped sql dump of our db is 2.1 GiB.
On our test instance (fork of the prod db from a few weeks ago) running the migration normally took 30-45 minutes.
For the production migration we disabled fsync (zfs set sync=disabled tank/lxc/foo) and it took 4m10.926s.