If developers generally don't run tests locally, is having arc unit do anything really valuable in your environment? That is, if you solved this by de-configuring arc so that arc unit did nothing, what would you lose?

(One possible thing you might lose is a consistent way for Harbormaster to execute tests in any project, but I'm not sure if you're using arc unit --target and, if you are, a possible solution might be to add an "only run tests when running in target mode" sort of flag rather than trying to change the behavior of --nounit.)

Yes we use arc unit as the chokepoint for "consistently run unit tests as far as phabricator is concerned", and use the --target flag when harbormaster is running the tests.

Developers do run the test suite locally if they suspect they probably broke things though, it's not "never", it's just "infrequently".

My general line of thinking is that providing a better idea of a "context" to arc unit (e.g., diff vs build) and letting it make a default decision about which tests to run (possibly including no tests) is probably the most promising way forward here. You could still override this context with some kind of arc diff --run-some-extra-tests flag. I'm not sure if hard-coding a small number of contexts or making them fully user-configurable or something in between is the best way forward. This is probably a longer-term thing.

Also in the mediumer-term you'll likely be able to disable this message through configuration with T6030.

In the short term, this warning message ("Unit tests were skipped...") currently examines only the arc unit tests, but we can make it examine arc + Harbormaster instead. Then it will say "skipped" only if there are no server-side tests. Additionally, it can accurately warn about Harbormaster build failures and caution users if builds are ongoing.

<?php

final class PhavNoUnitCustomArgument extends ICCustomArcanistArgument {

  public function supportsCommand($command) {
    return $command === 'diff';
  }

  public function getConfigurationKey() {
    return 'remote-unit';
  }

  public function getArguments() {
    return [
      'unit' => [
        'help' => pht(
          'Run the unit tests locally and attach the results to the diff.'),
      ],
    ];
  }

  public function willRunWorkflow($command, ArcanistWorkflow $workflow) {
    $passed = $workflow->getPassedArguments();
    if (in_array('--unit', $passed)) {
      return;
    }
    $nounit_arguments = [
      '--nounit',
      '--excuse',
      'remote tests',
    ] + $passed;
    $workflow->parseArguments($nounit_arguments);
  }

}

This works for client side stuff, our "custom argument class" just passes on the word from our custom ArcanistConfiguration.