0f67c89474
This reverts commit
|
||
|---|---|---|
| .. | ||
| bin | ||
| lib/src | ||
| test | ||
| .gitignore | ||
| analysis_options.yaml | ||
| pubspec.yaml | ||
| README.md | ||
| testing.json | ||
Gardening Tools
This directory is created for gathering all tools doing gardening in one place. Every tool or script, big or small should go here, and over time, hopefully we will have a useful collection of tools that support every part of the gardening work.
The current (working) tools are:
All the tools have been created in an add-hoc manner, thus they solve specific tasks that may not match your workflow fully. Feel free to add functionality to these. Below is a detailed description of each of the tools.
results
The results tool should be the primary tool for looking at failures and updating of status files. The tool fetches results.logs generated for each invocation of test.py on either the build bots or the CQ and matches the actual result for each test against the status files in your repository. Use it by calling:
dart results.dart get failures <argument>
The arguments can be one of the following:
get failures <result.log> : for a local result.log file.
get failures <uri_to_result_log> : for direct links to result.logs.
get failures <uri_try_bot> : for links to try bot builders.
get failures <commit_number> <patchset> : for links to try bot builders (see example below).
get failures <builder> : for a builder name.
get failures <builder> <build_number> : for a builder and build number.
get failures <builder_group> : for a builder group.
Some common workflows are listed below.
Finding failures on the CQ by link
If a CQ job fails then you will receive an email about try jobs failing. There is a direct link in the email to the builder that observed a failure:
https://ci.chromium.org/swarming/task/<swarm_task_id>?server=chromium-swarm.appspot.com
Copy the link and paste it in the command below to see all failures for that builder:
dart tools/gardening/bin/results.dart get failures <url>
All result logs fetched.
Calling test.py to find status files for the configuration and the expectation for 18298 tests. Estimated time remaining is 1 seconds...
FAILED: analyzer/test/generated/strong_mode_kernel_test
Result: RuntimeError
Expected: {Slow, Pass}
Configuration: mode=release, arch=x64, compiler=none, runtime=vm, checked, system=windows, use-sdk, builder-tag=win7
To run locally (if you have the right architecture and runtime):
tools/test.py --mode=release --arch=x64 --compiler=none --runtime=vm --checked --system=windows --use-sdk --builder-tag=win7 pkg/analyzer/test/generated/strong_mode_kernel_test
In the PolyGerrit interface under Tryjobs, for each try builder, you can obtain the link by right-clicking a builder-link and use copy link address.
Finding failures on the CQ by CL number and patch set
To get all failures from all builders in a try run, you can use the following command:
dart tools/gardening/bin/results.dart get failures <CL number> <patch set>
The CL number is shown at the top of the page, and in the URL, and the patch set number is shown in the try jobs pane.
Finding failures on the build bots
There are a few ways to find failures on build bots.
To find failures for an entire builder group use the following:
dart tools/gardening/bin/results.dart get failures <group>
The groups are shown on the main waterfall: vm,vm-kernel,analyzer...
To find failures for the latest build for a single build bot, use the following:
dart tools/gardening/bin/results.dart get failures <builder_name>
If you want to check failures on a specific build on a builder, use the build number:
dart tools/gardening/bin/results.dart get failures <builder_name> <build_no>
Remember, that statuses are tested against status files in your current repository. As a result, previous errors may not be reported, if the status files have changed.
You can find failures directly from a result log from a run of test.py. You can pass the URL linking to a result log, or you can pass the file name of a local result log on the command line.
dart tools/gardening/bin/results.dart get failures <url>
Finally, you can read the result log from a file by passing the file as so:
dart tools/gardening/bin/results.dart get failures <file>
Updating the status files
For now, the tool does not suggest any updates to status files, however, the tool can help to verify that changes to the status files has been done correctly.
If a build is failing, the results tool will show which tests failed and how their results differed from the expectations.
Update the status files appropriately, save the status files and run the tool again with the same arguments as before. If the changes are correct, the tool will now report that all test passes.
Note, that changed test files are not rerun, and new status annotations are not picked up
Generating result.log files for my own test runs
If you, for some reason, would like to have a result.log output for a test.py
invocation, just pass the flag --write-result-log to test.py. The default
output-directory is logs, therefore the file would be written to
logs/result.log. Be aware that existing files are overwritten.
To change the output directory, use the --output-directory option.
status_overlapping
If a test appears in multiple sections whose conditions are simultaneously true, the test's expectations is the union of the applicable entries. It's often unintended and undesirable to have overlapping sections for this reason. To find overlapping sections, run:
dart tools/gardening/bin/status_overlapping.dart <suite>
or
dart tools/gardening/bin/status_overlapping.dart <suite> <test>
Example output can be:
...
~/dart-sdk/sdk/tests/lib_2/lib_2_analyzer.status
5: [ $compiler == dart2analyzer ]
9: mirrors/deferred_mirrors_metadata_test: [Fail]
10: mirrors/deferred_type_test: [StaticWarning, OK]
11: mirrors/generic_f_bounded_mixin_application_test: [StaticWarning]
12: mirrors/mirrors_nsm_mismatch_test: [StaticWarning, OK]
13: mirrors/mirrors_nsm_test: [StaticWarning, OK]
14: mirrors/mirrors_nsm_test/dart2js: [StaticWarning, OK]
17: mirrors/repeated_private_anon_mixin_app_test: [StaticWarning, OK]
19: [ $compiler == dart2analyzer && $strong ]
21: mirrors/deferred_mirrors_metadata_test: [StaticWarning]
22: mirrors/deferred_type_test: [CompileTimeError, OK]
23: mirrors/generic_f_bounded_mixin_application_test: [CompileTimeError]
24: mirrors/mirrors_nsm_mismatch_test: [CompileTimeError, OK]
25: mirrors/mirrors_nsm_test: [CompileTimeError, OK]
26: mirrors/mirrors_nsm_test/dart2js: [CompileTimeError, OK]
28: mirrors/repeated_private_anon_mixin_app_test: [CompileTimeError, OK]
...
Notice that the sections overlap, but there may not be any relationship between entries in a section or file, ex. line 9 and 21. This is easier to see if overlapping entries are grouped by tests. To divide overlapping sections and entries into tests, use --print-test.
compare_failures
This tool compares a test log of a build step with previous builds. This is particularly useful to detect changes over time, such as flakiness of failures, timeouts or to find the build where a test started failing. The tool can be used for a single builder (vertically) or a builder-group (horizontally and vertically).
The tool displays the result of the latest 10 runs, which can be modified by the
option --run-count=X where X is the number of latest results to display.
Some typical use-cases are shown below.
Finding failing tests on one builder
Say a builder is going red, which means, for a step in that builder, some tests have not met their expectation. One can then find stdout log of that step and pass to the tool, as such:
Usage:
dart bin/compare_failures.dart <stdio-url>
where <stdio-url> is a url for a test log ("https://.../logs/stdio") from the
buildbot. Here is an example:
dart bin/compare_failures.dart https://uberchromegw.corp.google.com/i/client.dart/builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked%20vm%20tests/logs/stdio
Errors for /builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked vm tests :
none-vm-checked release_x64 standalone/io/http_server_response_test
6786: Pass / Crash
6785: / -- OK --
6784: / -- OK --
6783: / -- OK --
6782: / -- OK --
6781: / -- OK --
6780: / -- OK --
6779: / -- OK --
6778: / -- OK --
6777: / -- OK --
6776: / -- OK --
Finding failing tests on a group of builders
Sometimes it may be that a group of builders started to turn red. To find if there is any commonality between failing test, use the following command:
Usage:
dart bin/compare_failures.dart <build-group>
where <build-group> is the name of the build-group as shown in the console.
Here is an example:
dart bin/compare_failures.dart analyzer
Timeouts for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/strong_mode_driver_test
vm
7106: 0:00:21.086108
7105: 0:00:19.712000
7104: 0:01:00.774077
7103: 0:00:19.315931
7102: 0:00:21.380137
7101: 0:00:16.947695
7100: 0:00:13.226322
7099: 0:00:15.951636
7098: 0:00:19.665966
7097: 0:00:15.654565
7096: 0:00:24.775477
Errors for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test
7106: Pass Slow / RuntimeError
7105: Pass Slow / RuntimeError
7104: Pass Slow / RuntimeError
7103: / -- OK --
7102: / -- OK --
7101: / -- OK --
7100: / -- OK --
7099: / -- OK --
7098: / -- OK --
7097: / -- OK --
7096: / -- OK --
Errors for /builders/analyzer-win7-release-strong-be/builds/3716/steps/analyzer unit tests :
none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test
3716: Pass Slow / RuntimeError
3715: Pass Slow / RuntimeError
3714: Pass Slow / RuntimeError
3713: / -- OK --
3712: / -- OK --
3711: / -- OK --
3710: / -- OK --
3709: / -- OK --
3708: / -- OK --
3707: / -- OK --
3706: / -- OK --
No errors found for the 52 remaining bots.
status_summary
Collects the configurations for all status files in the 'tests' folder that mention one of the test names given as argument. This is useful to see what the expectation of a particular test/group of tests have on different configurations.
Usage:
dart bin/status_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test or function_subtype_typearg2_test.
Finding a summary for a specific test
Say that the test function_subtype_typearg2_test suddenly started to fail with
a RuntimeError on Safari in the dart2js safari builder group. It may be that
similar behavior was already spotted in the dart2js chrome builder group and had
been resolved (the status file was changed). One could then use run the
following command:
dart bin/status_summary.dart function_subtype_typearg2_test
function_subtype_typearg2_test
file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language/language_dart2js.status
Crash [ $compiler == dart2js && $dart2js_with_kernel && $host_checked ] NoSuchMethodError: The method 'hasSubclass' was called on null.
Crash [ $compiler == dart2js && $dart2js_with_kernel && $minified ] NoSuchMethodError: The method 'hasSubclass' was called on null.
file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language_strong/language_strong.status
RuntimeError [ $compiler == dartdevc && $runtime != none ] Issue 29920
The output would then indicate if there are other expectations for the test
function_subtype_typearg2_test in other configurations.
current_summary
Collects the test results for all build bots in [buildGroups] (defined in lib/src/buildbot_data.dart) for tests that mention one of the test names given as argument.
Usage:
dart bin/current_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test.
The results are currently pulled from the second to last build since the last build might not have completed yet.
Finding if a test fails on other configurations
Say that a test function_subtype_typearg2_test is behaving strange and a few
builders have started to turn red, and you suspect this particular test for
being responsible. Running the following will provide an answer:
dart bin/current_summary.dart function_subtype_typearg2_test
Fetching "/builders/vm-mac-debug-simdbc64-be/builds/-1/steps/vm tests" + 33 more ...
Fetching "/builders/app-linux-debug-x64-be/builds/-1/steps/vm tests" + 2 more ...
Fetching "/builders/vm-kernel-linux-release-x64-be/builds/-1/steps/front-end tests" + 7 more ...
Fetching "/builders/vm-win-debug-ia32-russian-be/builds/-1/steps/vm tests" + 11 more ...
Fetching "/builders/vm-noopt-simarm64-mac-be/builds/-1/steps/test vm" + 9 more ...
Fetching "/builders/vm-linux-product-x64-be/builds/-1/steps/vm tests" + 2 more ...
Fetching "/builders/vm-linux-debug-x64-reload-be/builds/-1/steps/vm tests" + 11 more ...
Fetching "/builders/dart2js-linux-d8-hostchecked-unittest-1-5-be/builds/-1/steps/dart2js-d8 tests" + 54 more ...
Fetching "/builders/dart2js-linux-d8-minified-1-5-be/builds/-1/steps/dart2js-d8 tests" + 79 more ...
Fetching "/builders/dart2js-linux-jsshell-1-4-be/builds/-1/steps/dart2js-jsshell tests" + 39 more ...
Fetching "/builders/analyzer-mac10.11-release-be/builds/-1/steps/analyze tests" + 53 more ...
Fetching "/builders/dart2js-linux-chromeff-1-4-be/builds/-1/steps/dart2js-chrome tests" + 79 more ...
Fetching "/builders/dart2js-linux-drt-1-2-be/builds/-1/steps/dart2js-drt tests" + 39 more ...
Fetching "/builders/dart2js-mac10.11-safari-1-3-be/builds/-1/steps/dart2js-safari tests" + 29 more ...
Fetching "/builders/dart2js-win8-ie11-1-4-be/builds/-1/steps/dart2js ie11 tests" + 111 more ...
Fetching "/builders/pkg-mac10.11-release-be/builds/-1/steps/package unit tests" + 5 more ...
language/function_subtype_typearg2_test
pass: none-vm vm-mac-debug-simdbc64-be/vm tests
pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests
...
language_2/function_subtype_typearg2_test
pass: none-vm vm-mac-debug-simdbc64-be/vm tests
pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests
pass: none-vm vm-mac-release-simdbc64-be/vm tests
pass: none-vm-checked vm-mac-release-simdbc64-be/checked vm tests
pass: none-vm vm-linux-debug-x64-be/vm tests
fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
...
The above output has been truncated (shown as ...), to better indicate how the result of a test is shown. One can then grep for failing tests, as so:
dart bin/current_summary.dart --group d8-minified language_2 | grep "fail:" -B 5 -A 5
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests
language_2/function_subtype_typearg2_test
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests
fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests
language_2/class_codegen_test
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
--
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests
pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests
...
luci
Luci is a tool made to query luci/logdog for information. The tool can be used to find the information about build-bots, build groups, build-details and commits across builds. There is nothing statically entered in the code files regarding the bots, thus this tool has all current information.
Usage:
dart bin/luci.dart <command>
To find help about each of the sub-tools, use help as <command>.
Find build-bots
The primary build-bots are those build-bots shown in the console view. To get them as list:
dart bin/luci.dart --build-bots
ddc-win-release-stable
pkg-linux-release-dev
dart2js-mac10.11-safari-3-3-be
vm-mac-debug-x64-be
vm-win-debug-ia32-russian-stable
vm-win-debug-ia32-be
dart2js-linux-drt-1-2-stable
dart-sdk-linux-be
pkg-win7-release-stable
vm-kernel-mac-release-x64-be
dart2js-linux-d8-hostchecked-unittest-5-5-stable
...
Similarly, all build bots can be found by:
dart bin/luci.dart --build-bots-all
Find build-groups
The build groups can be shown as such:
dart bin/luci.dart --build-groups
safari
vm
dart-sdk
vm-kernel
ddc
vm-product
analyzer
vm-misc
vm-reload
dart2js-jsshell
vm-app
pub-pkg
chrome
vm-precomp
dart2js-windows
dart2js-d8-minified
dart2js-linux
dart2js-d8-hostchecked
misc
This should match the view of the console.
Find builders in group
The tool can show all the builders in a specific group:
dart bin/luci.dart --builders-in-group <group>
where <group> is one of the groups shown above.
To find all the builders of the chrome builder group, use the following:
dart bin/luci.dart --builders-in-group chrome
dart2js-mac10.11-chrome-be
dart2js-linux-drt-1-2-be
dart2js-linux-drt-2-2-be
dart2js-linux-drt-csp-minified-be
Find detailed information about a build-bot
To find detailed information about a build on a build-bot, the following command can be used. This essentially gives a view similar to the html-page that shows information about a specific build, since it will give information about the steps, files and commits.
Usage:
dart bin/luci.dart --build-details <builder> <number>
where <builder> is the name of the builder and <number> is the build number.
It can be used in the following way:
dart bin/luci.dart --build-details vm-linux-debug-x64-reload-be 2971
Builds with commit
The buildbot assigns a build number to each build on a builder. To find the build numbers corresponding to a specific commit, use the --builds-with-commit option as follows:
Usage:
dart bin/luci.dart --builds-with-commit <commit_hash>
where <commit_hash> is the full hash of the specific commit. Since all bots
have to be checked, and one normally wants to investigate the latest commits,
the cache is only 15 minutes.
To run it, fx, on the hash 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c, one can
run the following:
dart bin/luci.dart --builds-with-commit 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c
2017-09-18 14:42:04.977786 Info: Sorry - this is going to take some time, since we have to look into all 25 latest builds for all bots for client client.dart.
Subsequent queries run faster if caching is not turned off...
The commit '4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c is used in the following builds:
dart2js-mac10.11-safari-3-3-be: #6821 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-3-3-be/6821
vm-mac-debug-x64-be: #13007 https://luci-milo.appspot.com/buildbot/client.dart/vm-mac-debug-x64-be/13007
vm-win-debug-ia32-be: #5270 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-be/5270
dart-sdk-linux-be: #15544 https://luci-milo.appspot.com/buildbot/client.dart/dart-sdk-linux-be/15544
vm-kernel-mac-release-x64-be: #1441 https://luci-milo.appspot.com/buildbot/client.dart/vm-kernel-mac-release-x64-be/1441
ddc-mac-release-be: #972 https://luci-milo.appspot.com/buildbot/client.dart/ddc-mac-release-be/972
vm-win-product-x64-be: #8433 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-product-x64-be/8433
analyze-linux-be: #3696 https://luci-milo.appspot.com/buildbot/client.dart/analyze-linux-be/3696
vm-win-debug-x64-be: #5096 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-x64-be/5096
vm-win-debug-ia32-russian-be: #3836 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-russian-be/3836
dart2js-mac10.11-safari-1-3-be: #6730 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-1-3-be/6730
...