Have you tried the test console?

what test console?

Derp. I'll check it out.

Ok so I just took it for a spin while editing herald rules. While it sort of solves my problems, it's very much a poor-man's solution - it's completely out of the way from the "create a herald rule" workflow - ideally, on the page where I'm creating or editing a herald rule, the test console should be right there so I can try out my new rule without necessarily having to save it; also it would filter to just trying that rule instead of trying all the rules, which would cut down on visual noise. This sort of change would make it much easier for the non-power user to create working herald rules.

Secondly, it doesn't totally solve my problem because I care as much about what sorts of diffs the rule matches as the ones that it doesn't match - and having to go searching for examples of diffs that it should and shouldn't match is potentially painful - it's not hard to verify it has the expected behavior on a diff or two, but what I really want to see is the "wait it matched THAT too?" surprises. We commit 100s of diffs a day so any false positives herald matches really spam my inbox, and having to wade through to figure out why herald added me to a diff, only to realize it's because my rule is overmatching, is not a very fun experience.

@epriestley should I open new tasks for these issues?

Is this rule a lint-like rule, in the vein of T7617? See also T10523 for discussion of lint-like rules.

I suppose the rules I am writing are fairly lint-like; part of this
reflects the sorry state of our lint infrastructure, though it is probably
a bit more flexible than herald. Having said that, sometimes I do want to
have rules that subscribe me to various diffs, before I figure out how to
automate my own advice; is there a way to add subscribers via lint rules?

Btw if Herald isn't supposed to be like lint, what is the intended use case?

is there a way to add subscribers via lint rules?

Not currently.

Btw if Herald isn't supposed to be like lint, what is the intended use case?

Here are some of the rules we have on this install:

H2, H11, and H77, and H95 send me email for every commit, revision, and push to any repository, and every wiki edit.

In the past, H124 automatically CC'd me on Ponder questions.

H104 CCs me on all tasks tagged with Phacility Support.

H65 adds "Blessed Reviewers" as a blocking reviewer for all revisions against open source repositories.

H68 prevents enormous commits to any hosted repository.

H78 triggers Harbormaster builds for revisions submitted by members of Community or Phacility High Command; H130 triggers builds for all open source repository commits.

H86 requires all contributors sign L28, the CLA.

Offhand, T8644 and T7804 describe a few more use cases.

All of these are complex behaviors which are particular to my personal workflow or this project's workflow and wouldn't make sense to encode as default behaviors in Phabricator, but none examine change content or operate in a lint-like way.

Rules like these are the intended use case for Herald, and the test console is fairly well-suited to testing rules like "CC me on any high priority task tagged with iOS" because you only need to test a handful of objects to be confident that the rule is working like you expect.

Thanks for all the examples - definitely makes sense that these behaviors are a bit of a different class than ones that actually examine file content.

I think my earlier suggestion, about having the test console in the herald create flow, is possibly a useful improvement - should I file a separate task?