Capture some conventions around writing tests

Change-Id: Idf23690d944eddfbd575bcb608d6949c10e7d1f3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/261442
Commit-Queue: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>

pkg/analyzer/doc/implementation/tests.md

@@ -0,0 +1,145 @@
+# Testing the analyzer
+
+## Test mechanics
+
+The analyzer uses the `test_reflective_loader` package for most of its tests.
+The `test_reflective_loader` package uses the `test` package to actually run the
+tests, but it provides a JUnit style mechanism for writing the tests.
+
+### Directory layout
+
+The tests, as expected, are in the top-level `test` directory. The structure of
+the directories within the `test` directory should match the structure of the
+`lib` directory whenever possible. The tests are in files whose name ends with
+`_test.dart`. This convention is used by the test runner on the bots to identify
+the files to be run, so a failure to follow this convention will cause the tests
+to not be run on the bots.
+
+For convenience, every directory in the `test` directory (including the `test`
+directory) contains a file named `test_all.dart`. That file isn't run on the
+bots, but can be run manually in order to run all the tests in the containing
+directory and all subdirectories.
+
+### Test file content
+
+Within a test file, the tests are defined in one or more classes. By convention,
+the class name should end with `Test`. The class must be annotated with
+`@reflectiveTest`.
+
+In order for the file to be executable, it must define a `main` method that
+looks something like the following, with one invocation of
+`defineReflectiveTests` for every reflective test class in the file:
+
+```dart
+void main() {
+  defineReflectiveSuite(() {
+    defineReflectiveTests(CompilationUnitImplTest);
+    // ...
+  });
+}
+```
+
+When the tests are run the test loader will reflect on the specified class
+(`CompilationUnitImplTest` in the example above) to find all the zero parameter
+instance methods whose name starts with 'test_'. These methods should have a
+return type of either `void` or `Future<void>`.
+
+There are a couple of useful annotations defined for test methods.
+
+- You can annotate a test with `@FailingTest()` to indicate that it is expected
+  to fail when run. This allows us to commit tests for bugs before working on a
+  fix for those bugs. The constructor has some optional parameters that allow
+  you to specify the reason for the failure and an issue URL.
+
+- You can annotate a test with `@SkippedTest()` if the test should not be run.
+  The constructor has the same optional parameters for specifying the reason and
+  an issue URL.
+
+- During development, you can mark one or more tests with `@soloTest` to cause
+  those tests to be the only tests that are run.
+
+### Test names
+
+Defining tests as methods on a class rather than as invocations of the `test`
+and `group` functions has the advantage that we can define common test utilities
+and share them across a large number of tests. To do that without classes would
+be much harder.
+
+But this style of test also has the disadvantage that we can't organize the
+tests into groups. In order to overcome this disadvantage we have adopted an
+uncommon naming convention for the test methods that combines the camel case and
+snake case conventions.
+
+Let's start with an example. Assume that we have a class named `ToSourceVisitor`
+that is a visitor with a separate method for every class of AST node, and that
+we want to test every method. Using `group` and `test`, we'd probably create a
+group for the class, then a subgroup for each visit method. For nodes with
+optional children, such as a list literal (where the `const` modifier and type
+arguments are both optional) you might have a group for literals with or without
+the type arguments, and then tests both with and without the modifier. In other
+words, you might end up with a structure like this:
+
+```dart
+void main() {
+  group('ToSourceVisitor', () {
+    group('visitListLiteral', () {
+      group('with type arguments', () {
+        test('with const', () { ... });
+        test('without const', () { ... });
+      });
+      group('without type arguments', () {
+        test('with const', () { ... });
+        test('without const', () { ... });
+      });
+    });
+  });
+}
+```
+
+In out tests, the top-level group is replaced by the class, which would be
+named `ToSourceVisitorTest`. The methods would all start with `test`, and each
+group's name would be converted to a camelCase identifier with groups being
+separated with an underscore. So the equivalent to the code above would be:
+
+```dart
+class ToSourceVisitorTest {
+  void test_visitListLiteral_withTypeArguments_withConst() { ... }
+
+  void test_visitListLiteral_withTypeArguments_withoutConst() { ... }
+
+  void test_visitListLiteral_withoutTypeArguments_withConst() { ... }
+
+  void test_visitListLiteral_withoutTypeArguments_withoutConst() { ... }
+}
+```
+
+### Test cases
+
+Most of our tests take a small piece of Dart code and test the behavior of some
+piece of functionality. In most cases the Dart code is required to be a whole
+compilation unit, though there are a few tests where only a snippet of code is
+required. We have a couple of conventions that, while not strictly enforced, are
+generally followed.
+
+Test code generally appears in a multi-line string, even when it would fit on a
+single line, with the text fully left justified, and with the closing quotes on
+a separate line. For example:
+
+```dart
+  test_final_noInitializer() async {
+    await assertNoErrorsInCode('''
+abstract class C {
+  abstract final int x;
+}
+''');
+  }
+```
+
+Test code should be kept as short as possible. Use short names and don't
+include code that isn't required in order to test what's being tested.
+
+Test code should generally follow best practices unless the deviation from best
+practices is a necessary part of what's being tested.
+
+Don't use a name with a special meaning, like `main`, unless it's important that
+you do so for the test.