Page MenuHomePhabricator

Updating to latest Phabricator caused feed corruption
Closed, ResolvedPublic

Description

I updated our install to the following versions yesterday:

phabricator 6f275ba1441f39fbd34ea454cf28c18ca016bab0 (Mon, Jun 20)
arcanist 2374403e8f80a8fb16cdc1ce2843f932632e8cf0 (Fri, Jun 17)
phutil 51c179b4c000de34d85a62188b5f9efb28afba73 (Fri, Jun 17)

...and ran the usual storage upgrades. After bringing everything back up two things happened:

  1. The feed now shows every event that ever happened over the past four years as occurring yesterday at 2:06PM (when I performed the upgrade).
  1. Presumably as a result, Phabricator emailed notices for every event that's ever happened in the project (it's a four year old project with a lot of activity, so that's...a lot of email).

So obviously I'd like to try and help track down that issue, so let me know what I can provide...does the storage migrator write logs anywhere?

More pressingly, for me anyway, what's the easiest way for me to clean the feed up? Is there an easy way to clear the feed entirely to get rid of those events?

Event Timeline

Are all of these events commits?

Are all of the commits reachable from tags or refs, but not from branch heads?

https://secure.phabricator.com/w/changelog/2016.25/#diffusion

I've never heard of anyone running into an issue like this, and there are no tools to clean it up.

Can you give us more information on how we can reproduce what you're seeing?

Are all of these events commits?

Yes, it appears that all of the events are commits

Are all of the commits reachable from tags or refs, but not from branch heads?

https://secure.phabricator.com/w/changelog/2016.25/#diffusion

That also seems to be the case. I wasn't expecting the project to have quite so many unreachable commits (the repo predates me by quite a while). So these are all unreachable commits that have now been imported?

So these are all unreachable commits that have now been imported?

That would be consistent with what you've reported.

If this is the cause, it should only happen this once.

If it's still ongoing, you may be able to use this command to put repositories back into "importing" mode so they don't publish events (this is the mode used when you first add a repository):

phabricator/ $ ./bin/repository mark-imported --mark-not-imported XYZ

...where XYZ is a repository monogram or callsign. You can get a list with bin/repository list. That may not completely silence things, but should make any remaining processing complete more quietly.

epriestley claimed this task.

Presuming this is resolved, but let me know if I'm missing anything.

Just wanted to chime in that I've just encountered this as well. Just did an upgrade, and hundreds (possibly thousands?) of old commits are somehow being reprocessed and are pushed to the feed. It's been processing for a couple of hours now, no idea how much work remains....

Assuming everything is OK with this happening and this is just noise, is there some way to clean this out of the feed after it's done? (Or even just clean the feed entirely.)

To build on what was originally suggested, you can have all your repos stop publishing events by running the following which will list all repos and put them back to importing mode so they don't publish events:

cd phabricator/
for repo in `./bin/repository list`
do
	./bin/repository mark-imported --mark-not-imported $repo
done