Page MenuHomePhabricator

Added some additional assertion methods.
ClosedPublic

Authored by joshuaspence on Mar 9 2014, 1:18 AM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, Aug 28, 10:37 PM
Unknown Object (File)
Wed, Aug 28, 9:42 PM
Unknown Object (File)
Sun, Aug 25, 10:16 PM
Unknown Object (File)
Thu, Aug 22, 6:46 PM
Unknown Object (File)
Thu, Aug 22, 6:46 PM
Unknown Object (File)
Thu, Aug 22, 6:45 PM
Unknown Object (File)
Thu, Aug 22, 6:28 PM
Unknown Object (File)
Tue, Aug 20, 5:48 AM

Details

Summary

There are quite a few tests in Arcanist, libphutil and Phabricator that do something similar to $this->assertEqual(false, ...) or $this->assertEqual(true, ...).

This is unnecessarily verbose and it would be cleaner if we had assertFalse and assertTrue methods.

Test Plan

I contemplated adding a unit test for the getCallerInfo method but wasn't sure if it was required / where it should live.

Diff Detail

Repository
rARC Arcanist
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

joshuaspence edited the test plan for this revision. (Show Details)
joshuaspence edited edge metadata.
joshuaspence updated this object.

There's a little bit of discussion about this in rARCc6b1f3f07. I think I don't have much ground to stand on on assertTrue(), and that there are compelling arguments for assertReached() instead of assertEqual(true, true) or assertTrue(true).

However, do we actually have any callsites for assertFalse()? I notice D8461 doesn't use the method, for instance. A quick grep of phabricator/ suggests that it wouldn't find much traction there, either.

Mostly, I want to avoid writing assertGreaterThanOrEqualTo() and assertArrayKeyExists() and such, since I think they're much less clear than having a smal set of assertion methods.

epriestley edited edge metadata.

Well, I guess there's a whole lot of assertFalse() in D8462. I liked never needing to read the assert part to figure out what the assertion meant, but I think my stance is peculiar and far outside what anyone working with this code is likely to expect, given that all other frameworks offer a rich library of assertion helpers.

This revision is now accepted and ready to land.Mar 9 2014, 2:22 AM
epriestley updated this revision to Diff 20069.

Closed by commit rARCc42bef4e2546 (authored by @joshuaspence, committed by @epriestley).

I disagree with you in that I think having assertX, assertY and assertZ is much more useful than writing all assertions using assertEquals.

I believe that the strongest use case against using assertEquals is when doing $this->assertEquals(true, $x instanceof SomeClass). Writing this using assertEquals provides a very useless assertion message if the assertion fails (unless, of course, we write a custom assertion message such as "$x is not an instance of SomeClass"). Having an assertInstanceOf`, for example, can provide a much more

The case for assertTrue and assertFalse is weaker than the case for assertInstanceOf, but is a first step.

I understand the argument conceptually, but I feel like historically I've spent more time in rich-assertion environments when reading tests (because of the added complexity associated with reading assertions) than I've saved when fixing tests (because of slightly richer error messages).

As written, I think these assertions are less useful than assertEqual() because they don't show you the value. If you do assertEqual(true, $variable) and $variable isn't true, assertEqual() prints out the value of $variable when it fails, while these new assertions don't! To fix the test, the first thing you'll likely have to do is go add a print_r() above the failing assertion, which you wouldn't have to do if you just used assertEqual(). They also raise the same error message that assertEqual() does, so there's no additional clarity there.

Obviously, that's fixable -- I'm just not convinced that rich assertions really provide any value.

The problem is that $this->assertEquals(true, $x > 0) will not assist with debugging because it will only tell you:

Expected: true
  Actual: false

which is not really telling you anything of value. So I don't think that the assertion messages from assertFalse and assertTrue are any less valuable, unless you are testing a return value instead of a boolean condition (i.e. $this->assertEquals(true, some_function_which_might_return_true(...)). In these cases, I have tried to leave the $this->assertEquals(true|false, ...) code as is,

A case where it's clearly less valuable is something like this:

assertEqual(false, get_thing(), 'Expect thing to not exist.');

With assertEqual(), you are told what get_thing() returned. With assertFalse(), you currently aren't, and your first step to fix the test will probably be adding print_r(get_thing()).

These two assertions specifically can never provide more information (although they can provide the same information with greater clarity), and their current implementations provide less.

Other assertions (like assertGreaterThanOrEqualTo()) can provide more information, but they also make tests harder to read. My argument is just I think that the cost of adding these rich assertions outweighs the modest gains from richer messages. I don't think there's any way to prove this, and I think this is a losing battle on my side, but I don't think it's self-evident or obviously true that adding 30 different flavors of assertion is necessarily a net win on test maintenance costs.

Just to be clear, in the example you provided ($this->assertEquals(false, get_thing()), I think that assertEqual should still be used. However, in the case of $this->assertEquals(false, array_key_exists($x, 'foo')) I feel that assertFalse is more concise whilst providing the same information.