Page MenuHomePhabricator

Obscure error when a UnitEngine returns wrong result type
Closed, ResolvedPublic

Description

A user reported the below:

I ran "arc diff HEAD~1 --trace" and got the following:
Running unit tests...
>>> [26] <exec> $ ./gradlew :core:clean :core:test --rerun-tasks
<<< [26] <exec> 23,751,727 us
[2015-06-29 22:32:56] ERROR 2: array_merge() expects at least 1 parameter, 0 given at [/Users/hannahintern/khan/android/.arcanist_extension/CommandTestEngine/src/CommandTestEngine.php:28]
arcanist(head=master, ref.master=55fbc331004b), java-test-engine(), phutil(head=master, ref.master=ea6830cc8019)
  #0 array_merge()
  #1 call_user_func_array(string, array) called at [<java-test-engine>/src/CommandTestEngine.php:28]
  #2 CommandTestEngine::parseOutput(integer) called at [<java-test-engine>/src/CommandTestEngine.php:10]
  #3 CommandTestEngine::run() called at [<arcanist>/src/wor
kflow/ArcanistUnitWorkflow.php:186]
  #4 ArcanistUnitWorkflow::run() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1342]
  #5 ArcanistDiffWorkflow::runUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1221]
  #6 ArcanistDiffWorkflow::runLintUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:475]
  #7 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:382]
[2015-06-29 22:32:56] ERROR 2: Invalid argument supplied for foreach() at [/Users/hannahintern/khan/devtools/arcanist/src/workflow/ArcanistUnitWorkflow.php:200]
arcanist(head=master, ref.master=55fbc331004b), java-test-engine(), phutil(head=master, ref.master=ea6830cc8019)
  #0 ArcanistUnitWorkflow::run() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1342]
  #1 ArcanistDiffWorkflow::runUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1221]
  #2 ArcanistDiffWorkflow::runLintUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:475]
  #3 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:382]
[2015-06-29 22:32:56] ERROR 2: Invalid argument supplied for foreach() at [/Users/hannahintern/khan/devtools/arcanist/src/workflow/ArcanistUnitWorkflow.php:268]
arcanist(head=master, ref.master=55fbc331004b), java-test-engine(), phutil(head=master, ref.master=ea6830cc8019)
  #0 ArcanistUnitWorkflow::run() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1342]
  #1 ArcanistDiffWorkflow::runUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1221]
  #2 ArcanistDiffWorkflow::runLintUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:475]
  #3 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:382]

[2015-06-29 22:32:56] EXCEPTION: (InvalidArgumentException) Argument 1 passed to mpull() must be of the type array, null given, called in /Users/hannahintern/khan/devtools/arcanist/src/workflow/ArcanistUnitWorkflow.php on line 285 and defined at [<phutil>/src/error/PhutilErrorHandler.php:200]
arcanist(head=master, ref.master=55fbc331004b), java-test-engine(), phutil(head=master, ref.master=ea6830cc8019)
  #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phutil>/src/utils/utils.php:107]
  #1 mpull(NULL, string) called at [<arcanist>/src/workflow/ArcanistUnitWorkflow.php:285]
  #2 ArcanistUnitWorkflow::run() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1342]
  #3 ArcanistDiffWorkflow::runUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:1221]
  #4 ArcanistDiffWorkflow::runLintUnit() called at [<arcanist>/src/workflow/ArcanistDiffWorkflow.php:475]
  #5 ArcanistDiffWorkflow::run() called at [<arcanist>/scripts/arcanist.php:382]

It looks like problem is that gradlew gave no output whatsoever (neither success nor failure). The command worked when she ran it again, so maybe it was a transient problem? In any case, the output was confusing -- maybe the code could notice when the user-specified test doesn't have expected output, and give a nicer error message in that case.

Event Timeline

csilvers raised the priority of this task from to Needs Triage.
csilvers updated the task description. (Show Details)
csilvers added a project: Arcanist.
csilvers added a subscriber: csilvers.

This looks like an error in custom third-party code you've written (CommandTestEngine), not the upstream.

Specifically, I'd guess:

  • CommandTestEngine isn't handling this case correctly, then calls array_merge() on something like null.
  • It returns what it expects to be an array, but isn't.
  • This causes a secondary error in the upstream code.

We could and should enforce this contract more directly by checking that the $results the engine returns are a list of test results, but that would just make the secondary error more specific ("Unit test engine 'CommandTestEngine' returned results in the wrong format from run(): expected a list of 'ArcanistUnitTestResult' objects.").

epriestley triaged this task as Normal priority.
epriestley renamed this task from Obscure error when a unittest runner gives no output to Obscure error when a UnitEngine returns wrong result type.Jun 30 2015, 2:24 PM