Drive-by mention in PHI91.

I realized I don't really understand the architecture of the notifications system or our deployment of it. Tell me what I got right:

From skimming the code in phabricator/support/aphlict and reading the docs, it looks like admin Aphlict servers are run locally on the same host as the Phabricator install, only bind to localhost by default, and receive (presumably) webhooks from that install. The client servers can run wherever, and accept WebSocket connections from actual clients. admin and client servers peer with each other to exchange messages, which flow from my.phab.install -> admin server -> client server -> browsers.

The notification.servers config option defines admin servers, which is where the Phabricator install will attempt to deliver freshly-baked notifications, and client servers, which are provided to browsers as the place they should be connecting to receive notifications.

For the Phacility cluster, the admin instance shares its notification configuration with the other Phacility-managed instances, which is the notify002 host and nlb001 load balancer. notify002 is bound to 0.0.0.0 for both client and admin servers, so there's presumably no reason we can't use that for delivering notifications from the admin instance.

I'm actually confused now about why we don't receive notifications on the admin instance. The Notification application appears to be installed, notification.servers appears correct, and the cluster config is the same as any other Phacility-hosted instance.

Servers

Mostly right: each node server process has both a "client" service and an "admin" service on different ports. Everything else is pretty much accurate, except that "admin server -> client server" is generally inside the same process.

I'm actually confused now about why we don't receive notifications on the admin instance.

Oh, maybe this is a lot closer to working than I thought. In fact, if you go to Settings → Notifications and click "Send Test Notification", it actually works already.

The reason we don't receive notifications is that Support doesn't emit them. To emit them, InstancesSupportIssueEditor needs to implement shouldPublishFeedStory() and return true. The individual InstancesSupportIssue...Transaction classes may also need some getTitleForFeed() implementations so the things the notifications say make sense.

To get the yellow "this page has been updated ,click to reload" bubble, InstancesSupportIssueViewController needs to call setPageObjectPHIDs() on the page it builds, so the UI "knows" that the page has the issue PHID.

This might cause /feed/ to turn into a bit of a mess (since we'll start publishing a lot of stories which almost no one can view, so the query may hit some policy check performance issues), but we can probably just uninstall the application if it does.

(Basically, a notification is just a copy of a feed story which we deliver directly to an individual at the same time we publish the feed story, so feed and notifications are intertwined.)

I think this is working properly in production, I recall getting at least one notification today.