Page MenuHomePhabricator

Modify the `lint-test` file format to allow for more powerful assertions
ClosedPublic

Authored by joshuaspence on Jan 3 2015, 8:00 AM.
Tags
None
Referenced Files
F13078982: D11176.id31715.diff
Tue, Apr 23, 7:53 PM
Unknown Object (File)
Sun, Apr 21, 9:29 PM
Unknown Object (File)
Sun, Apr 21, 3:37 AM
Unknown Object (File)
Fri, Apr 19, 2:12 AM
Unknown Object (File)
Fri, Apr 19, 2:12 AM
Unknown Object (File)
Fri, Apr 19, 2:12 AM
Unknown Object (File)
Fri, Apr 19, 2:12 AM
Unknown Object (File)
Fri, Apr 19, 2:12 AM
Subscribers

Details

Summary

Fixes T6854. The current format for lint-test files is somewhat inflexible and does not allow us to make assertions regarding the code or name of the linter messages (of class ArcanistLintMessage) that are raised. Specifically, the ${severity}:${line}:${char} format is hardcoded in ArcanistLinterTestCase. In this diff, I extend the this format to achieve the following goals:

  • Allow for the lint message code and name to be specified. Specifically, the full format is ${severity}:${line}:${char}:${code}:${name}.
  • Make all fields optional. error:3: will match any and all errors occuring on line 3.
  • Provide more useful output when assertions fail. Specifically, output all lint messages that are missing and/or surplus. Previously, only the first lint message was output.
Test Plan

arc unit

Diff Detail

Repository
rARC Arcanist
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

joshuaspence retitled this revision from to Modify the `lint-test` file format to allow for more powerful assertions.
joshuaspence updated this object.
joshuaspence edited the test plan for this revision. (Show Details)
joshuaspence added a reviewer: epriestley.
joshuaspence edited edge metadata.

Further progress

joshuaspence edited the test plan for this revision. (Show Details)

This is still a bit rough and can be optimized (because ArcanistLintResult::getMessages() returns a list<ArcanistLintMessage> roughly sorted by (line, char). I wanted to get some initial feedback before investing too much time in this.

Are "code" and "name" meaningfully different often enough to matter? The linters which don't define different codes are usually external linters, I think, and we don't have nearly as much of a burden to test them (e.g., our test suite does not need to cover all of the behavior of pylint or whatever, just that the binding operates correctly). The first-party linters where we do care about behavior (like XHPAST) have codes per message.

Specifically, could we simplify this by allowing the first field to either be a severity or a code, like error:3 vs TXT17:3? (Although this is a little fuzzy, it seems reasonable to expect that no linter will ever define "error" or "warning" as a code.)

Or could we just replace "severity" with "code" wholesale? That would be a bit of a pain to migrate, but could maybe just be automated.

I think this is generally headed to a good place, but that we might be able to end up with a simpler patch.

Are "code" and "name" meaningfully different often enough to matter?

Probably not, I just wasn't sure how far to take this.

Specifically, could we simplify this by allowing the first field to either be a severity or a code, like error:3 vs TXT17:3? (Although this is a little fuzzy, it seems reasonable to expect that no linter will ever define "error" or "warning" as a code.)

I don't see what that achieves, besides a slightly less verbose syntax? Longer term, I'd imagine that we'd have something more flexible such as severity=error|line=3|char=3|code=TXT1.

I should mention that one of the reasons that I want to pursue this is because I noticed that we aren't parsing the lint message correctly from lessc for ArcanistLesscLinter (although my current diff doesn't check the lint message text).

arc lint -- test.less 
>>> Lint for test.less:


   Error  (LESSCSyntaxError) Unknown lint message!
    Expected ')' got '('

               1 .a {
    >>>        2   something: (12 (13 + 5 -23) + 5);
               3 }

Possibly another plan of attack here would be to throw an exception if we can't parse the linter message.

epriestley edited edge metadata.
epriestley added inline comments.
src/lint/linter/__tests__/ArcanistLinterTestCase.php
193–203

Some possible issues with 0 as a code or line number, maybe.

This revision is now accepted and ready to land.Nov 23 2015, 4:07 PM
joshuaspence marked an inline comment as done.

Rebase

This revision was landed with ongoing or failed builds.May 21 2019, 3:48 AM
This revision was automatically updated to reflect the committed changes.