dart-sdk/tools/gardening

Gardening Tools

This directory is created for gathering all tools doing gardening in one place. Every tool or script, big or small should go here, and over time, hopefully we will have a useful collection of tools that support every part of the gardening work.

The current (working) tools are:

• results
• status_overlapping
• compare_failures
• status_summary
• current_summary
• luci

All the tools have been created in an add-hoc manner, thus they solve specific tasks that may not match your workflow fully. Feel free to add functionality to these. Below is a detailed description of each of the tools.

results

The results tool should be the primary tool for looking at failures and updating of status files. The tool fetches results.logs generated for each invocation of test.py on either the build bots or the CQ and matches the actual result for each test against the status files in your repository. Use it by calling:

dart results.dart get failures <argument>

The arguments can be one of the following:

    get failures <result.log>               : for a local result.log file.
    get failures <uri_to_result_log>        : for direct links to result.logs.
    get failures <uri_try_bot>              : for links to try bot builders.
    get failures <commit_number> <patchset> : for links to try bot builders (see example below).
    get failures <builder>                  : for a builder name.
    get failures <builder> <build_number>   : for a builder and build number.
    get failures <builder_group>            : for a builder group.

Some common workflows are listed below.

Finding failures on the CQ by link

If a CQ job fails then you will receive an email about try jobs failing. There is a direct link in the email to the builder that observed a failure:

https://ci.chromium.org/swarming/task/<swarm_task_id>?server=chromium-swarm.appspot.com

Copy the link and paste it in the command below to see all failures for that builder:

dart tools/gardening/bin/results.dart get failures <url>

All result logs fetched.
Calling test.py to find status files for the configuration and the expectation for 18298 tests. Estimated time remaining is 1 seconds...
	FAILED: analyzer/test/generated/strong_mode_kernel_test
	Result: RuntimeError
	Expected: {Slow, Pass}
	Configuration: mode=release, arch=x64, compiler=none, runtime=vm, checked, system=windows, use-sdk, builder-tag=win7

	To run locally (if you have the right architecture and runtime):
	tools/test.py --mode=release --arch=x64 --compiler=none --runtime=vm --checked --system=windows --use-sdk --builder-tag=win7 pkg/analyzer/test/generated/strong_mode_kernel_test

In the PolyGerrit interface under Tryjobs, for each try builder, you can obtain the link by right-clicking a builder-link and use copy link address.

Finding failures on the CQ by CL number and patch set

To get all failures from all builders in a try run, you can use the following command:

dart tools/gardening/bin/results.dart get failures <CL number> <patch set>

The CL number is shown at the top of the page, and in the URL, and the patch set number is shown in the try jobs pane.

Finding failures on the build bots

There are a few ways to find failures on build bots.

To find failures for an entire builder group use the following:

dart tools/gardening/bin/results.dart get failures <group>

The groups are shown on the main waterfall: vm,vm-kernel,analyzer...

To find failures for the latest build for a single build bot, use the following:

dart tools/gardening/bin/results.dart get failures <builder_name>

If you want to check failures on a specific build on a builder, use the build number:

dart tools/gardening/bin/results.dart get failures <builder_name> <build_no>

Remember, that statuses are tested against status files in your current repository. As a result, previous errors may not be reported, if the status files have changed.

You can find failures directly from a result log from a run of test.py. You can pass the URL linking to a result log, or you can pass the file name of a local result log on the command line.

dart tools/gardening/bin/results.dart get failures <url>

Finally, you can read the result log from a file by passing the file as so:

dart tools/gardening/bin/results.dart get failures <file>

Updating the status files

For now, the tool does not suggest any updates to status files, however, the tool can help to verify that changes to the status files has been done correctly.

If a build is failing, the results tool will show which tests failed and how their results differed from the expectations.

Update the status files appropriately, save the status files and run the tool again with the same arguments as before. If the changes are correct, the tool will now report that all test passes.

Note, that changed test files are not rerun, and new status annotations are not picked up

Generating result.log files for my own test runs

If you, for some reason, would like to have a result.log output for a test.py invocation, just pass the flag --write-result-log to test.py. The default output-directory is logs, therefore the file would be written to logs/result.log. Be aware that existing files are overwritten.

To change the output directory, use the --output-directory option.

status_overlapping

If a test appears in multiple sections whose conditions are simultaneously true, the test's expectations is the union of the applicable entries. It's often unintended and undesirable to have overlapping sections for this reason. To find overlapping sections, run:

dart tools/gardening/bin/status_overlapping.dart <suite>

or

dart tools/gardening/bin/status_overlapping.dart <suite> <test>

Example output can be:

...
~/dart-sdk/sdk/tests/lib_2/lib_2_analyzer.status
	5: [ $compiler == dart2analyzer ]
		9: mirrors/deferred_mirrors_metadata_test: [Fail]
		10: mirrors/deferred_type_test: [StaticWarning, OK]
		11: mirrors/generic_f_bounded_mixin_application_test: [StaticWarning]
		12: mirrors/mirrors_nsm_mismatch_test: [StaticWarning, OK]
		13: mirrors/mirrors_nsm_test: [StaticWarning, OK]
		14: mirrors/mirrors_nsm_test/dart2js: [StaticWarning, OK]
		17: mirrors/repeated_private_anon_mixin_app_test: [StaticWarning, OK]

	19: [ $compiler == dart2analyzer && $strong ]
		21: mirrors/deferred_mirrors_metadata_test: [StaticWarning]
		22: mirrors/deferred_type_test: [CompileTimeError, OK]
		23: mirrors/generic_f_bounded_mixin_application_test: [CompileTimeError]
		24: mirrors/mirrors_nsm_mismatch_test: [CompileTimeError, OK]
		25: mirrors/mirrors_nsm_test: [CompileTimeError, OK]
		26: mirrors/mirrors_nsm_test/dart2js: [CompileTimeError, OK]
		28: mirrors/repeated_private_anon_mixin_app_test: [CompileTimeError, OK]
...

Notice that the sections overlap, but there may not be any relationship between entries in a section or file, ex. line 9 and 21. This is easier to see if overlapping entries are grouped by tests. To divide overlapping sections and entries into tests, use --print-test.

compare_failures

This tool compares a test log of a build step with previous builds. This is particularly useful to detect changes over time, such as flakiness of failures, timeouts or to find the build where a test started failing. The tool can be used for a single builder (vertically) or a builder-group (horizontally and vertically).

The tool displays the result of the latest 10 runs, which can be modified by the option --run-count=X where X is the number of latest results to display.

Some typical use-cases are shown below.

Finding failing tests on one builder

Say a builder is going red, which means, for a step in that builder, some tests have not met their expectation. One can then find stdout log of that step and pass to the tool, as such:

Usage:

dart bin/compare_failures.dart <stdio-url>

where <stdio-url> is a url for a test log ("https://.../logs/stdio") from the buildbot. Here is an example:

dart bin/compare_failures.dart https://uberchromegw.corp.google.com/i/client.dart/builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked%20vm%20tests/logs/stdio

Errors for /builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked vm tests :
none-vm-checked release_x64 standalone/io/http_server_response_test
 6786:  Pass       / Crash
 6785:             / -- OK --
 6784:             / -- OK --
 6783:             / -- OK --
 6782:             / -- OK --
 6781:             / -- OK --
 6780:             / -- OK --
 6779:             / -- OK --
 6778:             / -- OK --
 6777:             / -- OK --
 6776:             / -- OK --

Finding failing tests on a group of builders

Sometimes it may be that a group of builders started to turn red. To find if there is any commonality between failing test, use the following command:

Usage:

dart bin/compare_failures.dart <build-group>

where <build-group> is the name of the build-group as shown in the console. Here is an example:

dart bin/compare_failures.dart analyzer

Timeouts for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/strong_mode_driver_test
         vm
 7106:   0:00:21.086108
 7105:   0:00:19.712000
 7104:   0:01:00.774077
 7103:   0:00:19.315931
 7102:   0:00:21.380137
 7101:   0:00:16.947695
 7100:   0:00:13.226322
 7099:   0:00:15.951636
 7098:   0:00:19.665966
 7097:   0:00:15.654565
 7096:   0:00:24.775477

Errors for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test
 7106:  Pass Slow  / RuntimeError
 7105:  Pass Slow  / RuntimeError
 7104:  Pass Slow  / RuntimeError
 7103:             / -- OK --
 7102:             / -- OK --
 7101:             / -- OK --
 7100:             / -- OK --
 7099:             / -- OK --
 7098:             / -- OK --
 7097:             / -- OK --
 7096:             / -- OK --

Errors for /builders/analyzer-win7-release-strong-be/builds/3716/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test
 3716:  Pass Slow  / RuntimeError
 3715:  Pass Slow  / RuntimeError
 3714:  Pass Slow  / RuntimeError
 3713:             / -- OK --
 3712:             / -- OK --
 3711:             / -- OK --
 3710:             / -- OK --
 3709:             / -- OK --
 3708:             / -- OK --
 3707:             / -- OK --
 3706:             / -- OK --

No errors found for the 52 remaining bots.

status_summary

Collects the configurations for all status files in the 'tests' folder that mention one of the test names given as argument. This is useful to see what the expectation of a particular test/group of tests have on different configurations.

Usage:

dart bin/status_summary.dart <test-name1> [<test-name2> ...]

where <test-nameX> are test names like language/arithmetic_test or function_subtype_typearg2_test.

Finding a summary for a specific test

Say that the test function_subtype_typearg2_test suddenly started to fail with a RuntimeError on Safari in the dart2js safari builder group. It may be that similar behavior was already spotted in the dart2js chrome builder group and had been resolved (the status file was changed). One could then use run the following command:

dart bin/status_summary.dart function_subtype_typearg2_test

function_subtype_typearg2_test
  file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language/language_dart2js.status
    Crash        [ $compiler == dart2js && $dart2js_with_kernel && $host_checked ] NoSuchMethodError: The method 'hasSubclass' was called on null.
    Crash        [ $compiler == dart2js && $dart2js_with_kernel && $minified ]     NoSuchMethodError: The method 'hasSubclass' was called on null.
  file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language_strong/language_strong.status
    RuntimeError [ $compiler == dartdevc && $runtime != none ]                     Issue 29920

The output would then indicate if there are other expectations for the test function_subtype_typearg2_test in other configurations.

current_summary

Collects the test results for all build bots in [buildGroups] (defined in lib/src/buildbot_data.dart) for tests that mention one of the test names given as argument.

Usage:

dart bin/current_summary.dart <test-name1> [<test-name2> ...]

where <test-nameX> are test names like language/arithmetic_test.

The results are currently pulled from the second to last build since the last build might not have completed yet.

Finding if a test fails on other configurations

Say that a test function_subtype_typearg2_test is behaving strange and a few builders have started to turn red, and you suspect this particular test for being responsible. Running the following will provide an answer:

dart bin/current_summary.dart function_subtype_typearg2_test

Fetching "/builders/vm-mac-debug-simdbc64-be/builds/-1/steps/vm tests" + 33 more ...
Fetching "/builders/app-linux-debug-x64-be/builds/-1/steps/vm tests" + 2 more ...
Fetching "/builders/vm-kernel-linux-release-x64-be/builds/-1/steps/front-end tests" + 7 more ...
Fetching "/builders/vm-win-debug-ia32-russian-be/builds/-1/steps/vm tests" + 11 more ...
Fetching "/builders/vm-noopt-simarm64-mac-be/builds/-1/steps/test vm" + 9 more ...
Fetching "/builders/vm-linux-product-x64-be/builds/-1/steps/vm tests" + 2 more ...
Fetching "/builders/vm-linux-debug-x64-reload-be/builds/-1/steps/vm tests" + 11 more ...
Fetching "/builders/dart2js-linux-d8-hostchecked-unittest-1-5-be/builds/-1/steps/dart2js-d8 tests" + 54 more ...
Fetching "/builders/dart2js-linux-d8-minified-1-5-be/builds/-1/steps/dart2js-d8 tests" + 79 more ...
Fetching "/builders/dart2js-linux-jsshell-1-4-be/builds/-1/steps/dart2js-jsshell tests" + 39 more ...
Fetching "/builders/analyzer-mac10.11-release-be/builds/-1/steps/analyze tests" + 53 more ...
Fetching "/builders/dart2js-linux-chromeff-1-4-be/builds/-1/steps/dart2js-chrome tests" + 79 more ...
Fetching "/builders/dart2js-linux-drt-1-2-be/builds/-1/steps/dart2js-drt tests" + 39 more ...
Fetching "/builders/dart2js-mac10.11-safari-1-3-be/builds/-1/steps/dart2js-safari tests" + 29 more ...
Fetching "/builders/dart2js-win8-ie11-1-4-be/builds/-1/steps/dart2js ie11 tests" + 111 more ...
Fetching "/builders/pkg-mac10.11-release-be/builds/-1/steps/package unit tests" + 5 more ...

language/function_subtype_typearg2_test
  pass: none-vm                      vm-mac-debug-simdbc64-be/vm tests
  pass: none-vm-checked              vm-mac-debug-simdbc64-be/checked vm tests
  ...
language_2/function_subtype_typearg2_test
  pass: none-vm                      vm-mac-debug-simdbc64-be/vm tests
  pass: none-vm-checked              vm-mac-debug-simdbc64-be/checked vm tests
  pass: none-vm                      vm-mac-release-simdbc64-be/vm tests
  pass: none-vm-checked              vm-mac-release-simdbc64-be/checked vm tests
  pass: none-vm                      vm-linux-debug-x64-be/vm tests
  fail: dart2js-d8                   dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
  ...

The above output has been truncated (shown as ...), to better indicate how the result of a test is shown. One can then grep for failing tests, as so:

dart bin/current_summary.dart --group d8-minified language_2 | grep "fail:" -B 5 -A 5

  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
  pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests
language_2/function_subtype_typearg2_test
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests
  fail: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
  pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests
language_2/class_codegen_test
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
--
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
  pass: dart2js-d8         dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
...

luci

Luci is a tool made to query luci/logdog for information. The tool can be used to find the information about build-bots, build groups, build-details and commits across builds. There is nothing statically entered in the code files regarding the bots, thus this tool has all current information.

Usage:

dart bin/luci.dart <command>

To find help about each of the sub-tools, use help as <command>.

Find build-bots

The primary build-bots are those build-bots shown in the console view. To get them as list:

dart bin/luci.dart --build-bots

ddc-win-release-stable
pkg-linux-release-dev
dart2js-mac10.11-safari-3-3-be
vm-mac-debug-x64-be
vm-win-debug-ia32-russian-stable
vm-win-debug-ia32-be
dart2js-linux-drt-1-2-stable
dart-sdk-linux-be
pkg-win7-release-stable
vm-kernel-mac-release-x64-be
dart2js-linux-d8-hostchecked-unittest-5-5-stable
...

Similarly, all build bots can be found by:

dart bin/luci.dart --build-bots-all

Find build-groups

The build groups can be shown as such:

dart bin/luci.dart --build-groups

safari
vm
dart-sdk
vm-kernel
ddc
vm-product
analyzer
vm-misc
vm-reload
dart2js-jsshell
vm-app
pub-pkg
chrome
vm-precomp
dart2js-windows
dart2js-d8-minified
dart2js-linux
dart2js-d8-hostchecked
misc

This should match the view of the console.

Find builders in group

The tool can show all the builders in a specific group:

dart bin/luci.dart --builders-in-group <group>

where <group> is one of the groups shown above.

To find all the builders of the chrome builder group, use the following:

dart bin/luci.dart --builders-in-group chrome

dart2js-mac10.11-chrome-be
dart2js-linux-drt-1-2-be
dart2js-linux-drt-2-2-be
dart2js-linux-drt-csp-minified-be

Find detailed information about a build-bot

To find detailed information about a build on a build-bot, the following command can be used. This essentially gives a view similar to the html-page that shows information about a specific build, since it will give information about the steps, files and commits.

Usage:

dart bin/luci.dart --build-details <builder> <number>

where <builder> is the name of the builder and <number> is the build number. It can be used in the following way:

dart bin/luci.dart --build-details vm-linux-debug-x64-reload-be 2971

Builds with commit

The buildbot assigns a build number to each build on a builder. To find the build numbers corresponding to a specific commit, use the --builds-with-commit option as follows:

Usage:

dart bin/luci.dart --builds-with-commit <commit_hash>

where <commit_hash> is the full hash of the specific commit. Since all bots have to be checked, and one normally wants to investigate the latest commits, the cache is only 15 minutes.

To run it, fx, on the hash 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c, one can run the following:

dart bin/luci.dart --builds-with-commit 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c
2017-09-18 14:42:04.977786 Info: Sorry - this is going to take some time, since we have to look into all 25 latest builds for all bots for client client.dart.
Subsequent queries run faster if caching is not turned off...
The commit '4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c is used in the following builds:
dart2js-mac10.11-safari-3-3-be: #6821	https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-3-3-be/6821
vm-mac-debug-x64-be: #13007	https://luci-milo.appspot.com/buildbot/client.dart/vm-mac-debug-x64-be/13007
vm-win-debug-ia32-be: #5270	https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-be/5270
dart-sdk-linux-be: #15544	https://luci-milo.appspot.com/buildbot/client.dart/dart-sdk-linux-be/15544
vm-kernel-mac-release-x64-be: #1441	https://luci-milo.appspot.com/buildbot/client.dart/vm-kernel-mac-release-x64-be/1441
ddc-mac-release-be: #972	https://luci-milo.appspot.com/buildbot/client.dart/ddc-mac-release-be/972
vm-win-product-x64-be: #8433	https://luci-milo.appspot.com/buildbot/client.dart/vm-win-product-x64-be/8433
analyze-linux-be: #3696	https://luci-milo.appspot.com/buildbot/client.dart/analyze-linux-be/3696
vm-win-debug-x64-be: #5096	https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-x64-be/5096
vm-win-debug-ia32-russian-be: #3836	https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-russian-be/3836
dart2js-mac10.11-safari-1-3-be: #6730	https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-1-3-be/6730
...