There are several classes of transaction like this. One is that all actions are restricted:

alincoln added a project: <restricted project>.

One is that some actions in a group of transactions are restricted:

alincoln added a project: <restricted project>.
alincoln added a comment:
Tagging this for the security team, since they should take a look before we release.

One is that some parts of some actions are restricted:

alincoln added projects: Security, Policy Review, <restricted project>.

And you could also have multiple transactions with a mixture of partially, fully, and unrestricted actions.

Transactions do not currently have any concept of being partially or fully restricted. That is, there is no $xaction->isPartiallyRestrictedToViewer($viewer); or $xaction->isFullyRestrictedToViewer($viewer); sort of method. It's possible to implement this, but likely not in a generic way, at least in all cases.

In cases where only some transactions are restricted or partially restricted, I think our behavior is probably correct already. It's consistent with what we do in other cases where a user makes type "A" and type "B" edits, but someone only wants notification about type "B" changes. For example, you get this mail:

alincoln added project: Skunkworks.
alincoln changed priority from "Low" to "High".

...if you are configured to receive either "project" changes or "priority" changes (or both). This behavior generally seems reasonable and aligns with expectation that most of the "cost" of the notification is in generating any notification at all, while the incremental cost of including less-interesting changes is relatively small.

In cases where all transactions are fully restricted, we could drop the mail without any real behavioral inconsistency. However, I'm hesitant about this because I think it's the kind of behavior that has the potential to be confusing.

You currently configure your mail preferences to say:

[ Email ] When a task's associated projects change.

The proposed behavior directly violates the language of the configuration setting.

This isn't insurmountable, but I'd like to have a much better story on T6297 first and have confidence that the vast majority of users are comfortable using tools we provide to understand and manage mail. I don't think we're there yet, and adding this as a weird special case puts us further from that goal.

Also, is this a problem on any install except this one? I'd expect installs other than this upstream to have relatively little restricted project/object activity on tasks -- I would normally expect the tasks to be completely hidden if they had sensitive components.

The restricted projects on this install are mostly corporate installs tracking state on private boards, but presumably few companies have secret teams which other teams aren't permitted to know about.

If this issue is unique to this install, one "fix" would be to require installs that want to track state in the upstream to do so in public.

We have started to encounter this as we use phabricator in a devops environment with many teams from a large organisation.

I received this e-mail recently:

eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.

TASK DETAIL
https://secure.phabricator.com/T1205

EMAIL PREFERENCES
https://secure.phabricator.com/settings/panel/emailpreferences/

To: eadler
Cc: marcus.hambraeus, stwalkerster, revi, devurandom, MZMcBride, eadler, jdloft, J5lx, Krenair, kAworu, aklapper, Afaque_Hussain, AnhNhan, qgil, chad, pixelpapst, garoevans, auduny, jevripio, hwinkel, mikaaay, btrahan, epriestley, davidreuss, joshuaspence, peter.schlaile, thz, Twilight, greggrossmeier, jeremyb, 20after4

In T10745#184113, @epriestley wrote:

[...]
And you could also have multiple transactions with a mixture of partially, fully, and unrestricted actions.
[...]
In cases where only some transactions are restricted or partially restricted, I think our behavior is probably correct already. It's consistent with what we do in other cases where a user makes type "A" and type "B" edits, but someone only wants notification about type "B" changes.

In cases where there's a single transaction and there is no visibility to the recipient, the current behavior of notifying a user seems plainly incorrect to me.

I guess I still have board changes notification e-mails enabled for this Phabricator installation because they are sometimes vaguely interesting (e.g., when a task I'm subscribed to gets moved to the "doing" column or whatever). But these e-mails where the entire substance of the message is something like "eadler added a project: Restricted Project." are starting to become irritating.

The support cost for the current behavior is very small. The behavior of the system is obvious. Users can file precise tasks like this one which clearly describe the problem (or find this task):

Emails are sent out about restricted changes

These kinds of issues are very easy for us to deal with (e.g., by spending a few seconds merging them here). They are also easy for users to deal with (they may be annoying, irritating, or frustrating, but they're easy to understand and the cost of getting extra mail is small).

The support cost for the proposed behavior (roughly: silently discard some mail) is potentially much larger. Based on other problems we see with users having difficulty understanding mail delivery, I believe users currently would often not have enough information to file tasks which point at the issue in detail. I would expect to see some reports like this instead:

Phabricator did not send email

This kind of report takes a much larger amount of time to respond to because I have to interact with the user and ask a series of diagnostic questions to figure out that the missing part of the report is "...about changes which only affect restricted objects". The user often won't have enough information to really rule anything out, and we'll have to look at their entire mail pipeline for answers.

Of course, these reports would be less frequent, but they take far, far longer to respond to.

We are resource-constrained, and don't get paid for support. When there's product tension between an irritating-but-obvious "positive" behavior and a nicer-but-confusing "negative" behavior, we usually have a strong incentive to make Phabicator's behavior obvious to reduce our support costs, even if that also means it will be annoying.

I understand that this is irritating and frustrating. We'd love to have the resources to provide more free support and have a wider margin of error to accept increased support costs on product choices like this, but we currently do not. We have a total of two employees. Both of us are unsalaried and have been for years, and we already spend a very large amount of time supporting Phabricator.

Beyond professional costs, support is not personally rewarding. Many mornings I wake up to a queue of reports from users who haven't bothered to read the instructions asking for help. I have done this essentially every day for about five years, and there is no end in sight.

When we have the opportunity to reduce support costs, we must make choices aggressively to do so.

There is a third path here, which is to improve the tools around understanding mail delivery so that users are more successful at answering mail questions themselves more often, and I can ask for less diagnostic information when we do receive vague reports. This would give us a wider margin of error and let us choose the less annoying behavior with less support risk.

This is captured in tasks like T6297, T9161, etc. We have been successful with this in the past in other arenas, and now pay a very low support cost for many problems (particularly installation and configuration) which we previously paid a high cost for but which Phabricator now can solve automatically for almost all users.

However, particularly for systems like mail, this is very complex. Many components of the mail pipeline can not be observed directly for product reasons (we can't let you see other users' mail or password reset tokens) or technical reasons (we have little ability to observe activity in third-party MTAs). Mail itself is complex, and we have very limited tools to tackle many mail problems.

For example, a possible solution to this problem might be to add an "X-Everything-In-This-Message-Is-Restricted" header to this mail. This is a solution with a relatively small technical cost and likely a small support cost: more advanced users could write a mail rule to discard it if they found it annoying, but newer users would never be faced with a potentially confusing behavior. But we can't do this because use of Gmail is international galactic law, and Gmail can't filter on X-headers.

I am confident that we will eventually improve these tools, but this is a very large problem that is difficult to prioritize because it has limited/unclear value and a large cost relative to other projects we can spend resources on instead.

If you have suggestions about how we can improve any of this, we're open to them. There is some additional discussion in T9212. But we simply don't have the resources to help every user with every problem they run into for free, and never will. We must choose how to allocate the limited resources we do have, and this kind of issue seems like the sort of thing that shouldn't be at the top of the priority list.

These types of e-mails are still being sent. Here's one from July 10, 2017:

epriestley added 1 revision(s): Restricted Differential Revision.

TASK DETAIL
https://secure.phabricator.com/T12681

EMAIL PREFERENCES
https://secure.phabricator.com/settings/panel/emailpreferences/

To: epriestley
Cc: alexmv, wienczny, gou1, pouyana, amckinley, urzds, johnny-bit, joshuaspence, nnayudu, aklapper, nemobis, greggrossmeier, bizrad6, jmeador, NDKilla, MZMcBride, richardvanvelzen, 20after4, cspeckmim, tomekj2ee, benwick, avivey, eadler, chad, epriestley, kuma-guy

Perhaps I'll write a mail rule to filter on the body contents.

I'm not sure I follow what you're trying to tell us.

In T10745#229028, @chad wrote:

I'm not sure I follow what you're trying to tell us.

Yeah, I know the feeling. That's kind of my point with these e-mails from Phabricator.

I've received maybe a half-dozen or more of these "Restricted Differential Revision" e-mails lately from T12681. After deleting them as the junk that they are, I remembered that there was a task about this. Phabricator search, being what it is, made it take a few tries to find this task, but here we are now, about a year after my previous comment. I've included the newest e-mail body contents in the hopes that future searches might be faster.

Personally, I continue to think that it would be very nice if Phabricator didn't spam people about changes they do not have access to. I find it rude that subscribing to a task results in these "Restricted Differential Revision" e-mails. Since filtering these e-mails server-side seems to be too difficult, I'm considering filtering them client-side instead.

Hmm, what did you try for search, off hand, do you recall?

FWIW, I don't think these are "pure" spam emails, as in, that they are 100% unwanted by everyone. Even if you cannot see the underlying diff, you've asked to be notified when work happens essentially on a task. I think it's difficult to have Phabricator assess whether this email is informative or not to each individual user who subscribed. To some who are waiting for the product, it's valuable, to you or others perhaps not at all. Client side filtering still seems like the correct workaround until T5791 (which is server side email filtering).

I'm sort of surprised T5791 has not been mentioned here reading back, this seems like a good fit.