I did read and take to heart your article on exit codes. I am however faced with a tricky situation in that the exit code used for an unsound test is 1 in ArcanistUnitWorkflow.php.
The issue that we are running into specifically is that we have a harbormaster build step that runs arc unit. In this step we rely on the exit status of arc to determine if it passed or failed. In practice this means that a status of 0 or 1 (SUCCESS or UNSOUND) means everything is "fine" and a status of anything else means it failed. however in doing some testing today I noticed that it's also possibly for arc to blow up before even getting to running the unit tests in which case all the unit tests get skipped and we have no way to distinguish this output or status code from that of an unsound test since a php fatal error exits with an exit code of one as well.
Here is an example:
Exception 2 Command failed with error #1! 3 COMMAND 4 '/var/drydock/workingcopy-3399/repo/super/secret/sauce' -c '/var/drydock/workingcopy-3399/repo/super/secret/sauce/something.yml' --manifest 5 6 STDOUT 7 (empty) 8 9 STDERR 10 Traceback (most recent call last): 11 File "/var/drydock/workingcopy-3399/repo/super/secret/sauce", line 724, in <module> 12 sys.exit(main()) 13 File "/var/drydock/workingcopy-3399/repo/super/secret/sauce/run_unit_tests", line 694, in main 14 print json.dumps(generate_manifest(), indent=2) 15 File "/var/drydock/workingcopy-3399/repo/super/secret/sauce/run_unit_tests", line 635, in generate_manifest 16 for test in discoverTests(): 17 File "/var/drydock/workingcopy-3399/repo/super/secret/sauce/run_unit_tests", line 276, in discoverTests 18 for filename in sorted(os.listdir(unitTestDir)): 19 OSError: [Errno 2] No such file or directory: '/var/drydock/workingcopy-3399/repo/super/secret/sauce/bin/tests' 20 21 (Run with `--trace` for a full exception trace.)
It seems like in this case error codes are already being used to signal the status of the command and having unsound overlap with a catastrophic failure could be problematic.