diff --git a/pkg/analyzer/doc/implementation/tests.md b/pkg/analyzer/doc/implementation/tests.md new file mode 100644 index 00000000000..56e3dd1bbb9 --- /dev/null +++ b/pkg/analyzer/doc/implementation/tests.md @@ -0,0 +1,145 @@ +# Testing the analyzer + +## Test mechanics + +The analyzer uses the `test_reflective_loader` package for most of its tests. +The `test_reflective_loader` package uses the `test` package to actually run the +tests, but it provides a JUnit style mechanism for writing the tests. + +### Directory layout + +The tests, as expected, are in the top-level `test` directory. The structure of +the directories within the `test` directory should match the structure of the +`lib` directory whenever possible. The tests are in files whose name ends with +`_test.dart`. This convention is used by the test runner on the bots to identify +the files to be run, so a failure to follow this convention will cause the tests +to not be run on the bots. + +For convenience, every directory in the `test` directory (including the `test` +directory) contains a file named `test_all.dart`. That file isn't run on the +bots, but can be run manually in order to run all the tests in the containing +directory and all subdirectories. + +### Test file content + +Within a test file, the tests are defined in one or more classes. By convention, +the class name should end with `Test`. The class must be annotated with +`@reflectiveTest`. + +In order for the file to be executable, it must define a `main` method that +looks something like the following, with one invocation of +`defineReflectiveTests` for every reflective test class in the file: + +```dart +void main() { + defineReflectiveSuite(() { + defineReflectiveTests(CompilationUnitImplTest); + // ... + }); +} +``` + +When the tests are run the test loader will reflect on the specified class +(`CompilationUnitImplTest` in the example above) to find all the zero parameter +instance methods whose name starts with 'test_'. These methods should have a +return type of either `void` or `Future`. + +There are a couple of useful annotations defined for test methods. + +- You can annotate a test with `@FailingTest()` to indicate that it is expected + to fail when run. This allows us to commit tests for bugs before working on a + fix for those bugs. The constructor has some optional parameters that allow + you to specify the reason for the failure and an issue URL. + +- You can annotate a test with `@SkippedTest()` if the test should not be run. + The constructor has the same optional parameters for specifying the reason and + an issue URL. + +- During development, you can mark one or more tests with `@soloTest` to cause + those tests to be the only tests that are run. + +### Test names + +Defining tests as methods on a class rather than as invocations of the `test` +and `group` functions has the advantage that we can define common test utilities +and share them across a large number of tests. To do that without classes would +be much harder. + +But this style of test also has the disadvantage that we can't organize the +tests into groups. In order to overcome this disadvantage we have adopted an +uncommon naming convention for the test methods that combines the camel case and +snake case conventions. + +Let's start with an example. Assume that we have a class named `ToSourceVisitor` +that is a visitor with a separate method for every class of AST node, and that +we want to test every method. Using `group` and `test`, we'd probably create a +group for the class, then a subgroup for each visit method. For nodes with +optional children, such as a list literal (where the `const` modifier and type +arguments are both optional) you might have a group for literals with or without +the type arguments, and then tests both with and without the modifier. In other +words, you might end up with a structure like this: + +```dart +void main() { + group('ToSourceVisitor', () { + group('visitListLiteral', () { + group('with type arguments', () { + test('with const', () { ... }); + test('without const', () { ... }); + }); + group('without type arguments', () { + test('with const', () { ... }); + test('without const', () { ... }); + }); + }); + }); +} +``` + +In out tests, the top-level group is replaced by the class, which would be +named `ToSourceVisitorTest`. The methods would all start with `test`, and each +group's name would be converted to a camelCase identifier with groups being +separated with an underscore. So the equivalent to the code above would be: + +```dart +class ToSourceVisitorTest { + void test_visitListLiteral_withTypeArguments_withConst() { ... } + + void test_visitListLiteral_withTypeArguments_withoutConst() { ... } + + void test_visitListLiteral_withoutTypeArguments_withConst() { ... } + + void test_visitListLiteral_withoutTypeArguments_withoutConst() { ... } +} +``` + +### Test cases + +Most of our tests take a small piece of Dart code and test the behavior of some +piece of functionality. In most cases the Dart code is required to be a whole +compilation unit, though there are a few tests where only a snippet of code is +required. We have a couple of conventions that, while not strictly enforced, are +generally followed. + +Test code generally appears in a multi-line string, even when it would fit on a +single line, with the text fully left justified, and with the closing quotes on +a separate line. For example: + +```dart + test_final_noInitializer() async { + await assertNoErrorsInCode(''' +abstract class C { + abstract final int x; +} +'''); + } +``` + +Test code should be kept as short as possible. Use short names and don't +include code that isn't required in order to test what's being tested. + +Test code should generally follow best practices unless the deviation from best +practices is a necessary part of what's being tested. + +Don't use a name with a special meaning, like `main`, unless it's important that +you do so for the test.