Can you give some better insight on how you use Herald? I'd like to understand the core problem (hard to find rule X) vs. just randomly adding features that might not solve the problem.

Currently we have over 1500 active herald rules. Over 1000 of them are global.

Some rules triggers builds on Jenkins. Some rules assigns responsible persons/ccs/projects for tasks. Most rules are global and we allow our users modify them.

For example we have custom field called "Team" on Maniphest with almost 500 items. Task owners, cc persons, projects can be added depending on Team field. But if you'd decide modify team, how to find proper rule which needs to be modified? We have naming patterns - in this case it is "Team: Some team name". If you know team name, it is possible find the rule. But current herald search does not have ability to search by name. At the best case I can filter all global maniphest rules which are enabled and will get 500 rows. Then perform search with CTRL+f on browser :-)

In my opinion very few phabricator setups has such amount rules as ours. And 2000-5000 records in single table is not much, so we can go with simple WHERE name LIKE '%query%'.

@chad ^^

In many cases, we must run Herald rules synchronously in the same process performing a write which triggers the rules. In some cases, this is strictly required: for example, when you push a commit, we clearly must evaluate at least some rules before we accept it, because a rule may say "don't accept this commit". If a rule may cause a "don't accept this commit" effect, we obviously must evaluate it before we accept the commit.

In other cases this is theoretically not required as strictly, but still very desirable: we want to apply Herald effects at the same time as we apply edits, so that after you create or update an object you can immediately see what Herald did to it and you only get one email about User + Herald changes.

Because this is required in some cases and desirable in all cases, we currently do it in all cases. This means that every additional rule imposes a small performance cost for every write. Eventually, you will reach a point where evaluating rules is expensive enough that the user experience gets really bad, since they have to wait for a long time every time they create or update a task. I'm curious how long rule evaluation is currently taking -- you can get a rough estimate of cost in Herald → Transcripts. The "27ms" (for example) shows the rule evaluation cost in Herald for that action:

If this shows a value like 1,000ms, that means every update to a task is taking an extra second in Herald before we can send a response back to the user. It's possible that Herald is performing better than I imagine, but I would guess that this cost is already in the hundreds of milliseconds at the very least.

A second concern is universal access to Herald rules. Global rules bypass policies. Although they aren't completely able to ignore policies after T6367 because their effects are still indirectly covered by policies, a clever user is likely to be able to achieve meaningful policy bypass either now or in the future. By giving all users access to Global rules, you are effectively preventing policies from being used for access control in a meaningful way. (Of course, maybe this is fine on your install.)

Because of the largely unavoidable performance cost of having a huge number of rules and the bypass behavior of having global rules, we discourage installs from writing thousands of global rules, and are hesitant to build features which make it easier to do this or which are only useful if you've done this or want to do this. The request here seems to fall into this category. Particularly, this can't scale forever -- do you have a plan for when you reach 5,000 rules or 10,000 rules and the evaluation cost is 5x or 10x higher than it is today? Or when you find a use case for policies and need to get rid of (or at least review) 4,900 rules written by users who should not have universal access?

In theory, we may be able to improve the performance of Herald rule evaluation when this day comes, but there are no needless, large inefficiencies today and I'd be surprised if we can make them evaluate more than 2x or 3x faster -- and even that is optimistic. We also don't currently plan to pursue this and broadly think of this as an unsupported use case (see " A turing-complete simulation of Conway's Game of Life implented in 958,000 Herald rules executes slowly." in Troubleshooting Performance Problems), so upstream support here will be begrudging and expensive if you eventually need it.

If many of your rules amount to "examine the 'team' field, then add projects, subscribers and owners", you can reduce all of these rules to one rule like this:

• Implement a "Teams" tool which defines your 500 teams and allows users to configure which owners/subscribers/projects each team should get implicitly.
• Write a custom "Apply effects from the Teams tool" Herald action.
• Write one Herald rule, like this:

When:
[ Always ]
Take these actions:
[ Apply effects from the Teams tool ]

This is more work upfront, but if you continue using Herald as a general-purpose business logic engine you will outscale it in much worse ways than this sooner or later, and our ability to make it more scalable is fundamentally limited by the need to evaluate rules in-process before sending a response to the user.

One note is that the transcript cost only includes the cost to evaluate the rules, not the cost to load the rules. The intent of the field was to catch rules which were unusually expensive to evaluate, but in the case of a large number of very simple rules, I would expect that the cost to load and build objects for rules would dominate the cost to evaluate them.

ApplicationEditor (T9132) and Nuance (T8783) are both scheduled for implementation before the end of the year and might offer alternatives here, too. With ApplicationEditor you could create 500 "create task" forms (instead of 500 teams) and have each form prepopulate with appropriate assignees, subscribers, projects, etc. With Nuance, you could create 500 queues (instead of 500 teams) and give each queue custom routing rules (and potentially have them just create tasks directly and not actually use Nuance).

Neither of these have the same inherent scaling challenges that Herald does, and we'll be able to handle many millions of them without any issues.

Currently we have over 1500 active herald rules. Over 1000 of them are global.
Some rules triggers builds on Jenkins. Some rules assigns responsible persons/ccs/projects for tasks. Most rules are global and we allow our users modify them.

Roughly what fraction are for builds?

T9352 has an early approach for parameterizing builds via Herald which might let you collapse some of those rules.

But see also T9519: my general expectation is that parameterization should be happening in repositories as much as possible, and that projects an organization works on should, in general, have a consistent ruleset for building and running tests in any environment (i.e., approximately some universal version of arc unit, whether that is actually arc unit or not) which is part of the repository and versions alongside the rest of the codebase.

That is, if you're telling a human how to run the tests, I expect that it is universally desirable for the instructions to be as simple as possible and share as much code with automated test runs as possible. I also expect humans to run tests regularly and to easily be able to run tests (for example, to debug or fix issues with the tests or test harness).

I'm open to exploring ways to make it easier to accomplish this, but I don't currently understand why a development environment might want 1,000 different ways to run tests, encoded in a machine-only format outside the repository. My belief is that this implies a corresponding wiki page somewhere with 1,000 walkthroughs for humans, or 1,000 such pages, or no human ability to run tests directly, all of which seem quite bad to me.