mirror of
https://github.com/dart-lang/sdk
synced 2024-07-19 20:17:27 +00:00
f0ca213d60
This CL optimizes how CFE et al presubmits are run. In the examples below we'll that it takes the presubmit time from 31+ to ~13 seconds, from 31+ to ~20 seconds and from 30+ to ~19 seconds on a few simple cases and from 76+ to ~27 seconds in a case where files in both _fe_analyzer_shared, front_end, frontend_server and kernel are changed. Before this CL, if there was changes in both front_end and frontend_server for instance it would run one smoke-test for each. They would each technically only test things in their own directory, but they would do a lot of overlapping work, e.g. compiling frontend_server also compiles front_end; the startup cost of a script is done several times etc. The bulk of the change in this CL is thus to only run things once. Now, if there is a change in both front_end and frontend_server the python presubmit will still launch a script for each, but it's just a light-weight script that will take ~400 ms to run (on my machine) if it decides to not do anything. What it does is that it looks at the changed files, from that it will know which presubmits will be run and decide which of them will actually do the work - the rest will just exit and say "it will be tested by this other one". Furthermore it then tries to run only the smoke tests necessary. For instance, if you have only changed a test in front_end it will only run the spell checker (and only for that file). Note that this is not perfect and there can be cases where you should get a presubmit error but wont. For instance if you remove all content from the spellchecking dictionary file it should give you lots of spelling mistake errors, but it won't because it won't actually run the spell checker (as no files it should spell check was changed). Probably you have to actively try to cheat it though, so I don't see it as a big problem. Things will still be checked fully on the CI. Additionally * the generated messages will have trailing commas which speeds up formatting of the generated files (in the cases where the generated files will have to be checked). * the explicit creation testing tool will do the outline of everything, but only do the bodies of the changed files. * building the "ast model" only compiles the outline. Left to do: * If only changing a single test, for instance, it will only run the spell checker on that file, but launching the isolate its run in still takes ~7 seconds because it loads up other stuff too. Maybe we could have special entry points for cases where it only should run an otherwise simple test. * The presubmit in the sdk dir (not CFE related) doesn't do well with many (big) changed files and testing them for formatting errors can easily take 10+ seconds (see example below where it contributes ~5 seconds for instance). Maybe `dart format` could be made faster, or maybe the script should test more than one file at once. *Example runs before and after*: Change in a single test file in front_end ========================================= Now: ``` $ time git cl presubmit -v -f [I2024-01-25 09:46:08,391 187077 140400494405504 presubmit_support.py] Found 1 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py Presubmit checks took 11.5s to calculate. Python 3 presubmit checks passed. real 0m12.772s user 0m16.093s sys 0m2.146s ``` Before: ``` $ time git cl presubmit -v -f [I2024-01-25 10:07:08,519 200015 140338735470464 presubmit_support.py] Found 1 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py 28.3s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Presubmit checks took 30.0s to calculate. Python 3 presubmit checks passed. real 0m31.396s user 2m9.500s sys 0m11.559s ``` So from 31+ to ~13 seconds. --------------------------------------------------------------------- Change in a single test file and a single lib file in front_end =============================================================== Now: ``` $ time git cl presubmit -v -f Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py 15.9s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Presubmit checks took 18.0s to calculate. Python 3 presubmit checks passed. real 0m19.365s user 0m33.157s sys 0m5.049s ``` Before: ``` $ time git cl presubmit -v -f [I2024-01-25 10:08:36,277 200953 140133274818432 presubmit_support.py] Found 2 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py 27.9s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Presubmit checks took 30.0s to calculate. Python 3 presubmit checks passed. real 0m31.311s user 2m9.854s sys 0m11.898s ``` So from 31+ to ~20 seconds. --------------------------------------------------------------------- Change only the messages file in front_end (but with generated files not changing) ================================================================================== Now: ``` $ time git cl presubmit -v -f [I2024-01-25 09:53:02,823 190466 140548397250432 presubmit_support.py] Found 1 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py 15.6s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Presubmit checks took 17.0s to calculate. Python 3 presubmit checks passed. real 0m18.326s user 0m38.999s sys 0m4.530s ``` Before: ``` $ time git cl presubmit -v -f [I2024-01-25 10:10:04,431 201892 140717686302592 presubmit_support.py] Found 1 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/front_end/PRESUBMIT.py 28.0s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Presubmit checks took 29.2s to calculate. Python 3 presubmit checks passed. real 0m30.550s user 2m9.488s sys 0m11.689s ``` So from 30+ to ~19 seconds. --------------------------------------------------------------------- Change several files: ``` $ git diff --stat pkg/_fe_analyzer_shared/lib/src/messages/codes_generated.dart | 4 ++-- pkg/_fe_analyzer_shared/lib/src/parser/listener.dart | 2 ++ pkg/front_end/lib/src/api_prototype/incremental_kernel_generator.dart | 2 ++ pkg/front_end/lib/src/base/processed_options.dart | 2 ++ pkg/front_end/messages.yaml | 2 +- pkg/front_end/tool/dart_doctest_impl.dart | 2 ++ pkg/frontend_server/lib/compute_kernel.dart | 2 ++ pkg/kernel/lib/ast.dart | 2 ++ 8 files changed, 15 insertions(+), 3 deletions(-) ``` ==================== Now: ``` [I2024-01-25 09:57:53,270 193911 140320429016960 presubmit_support.py] Found 8 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/_fe_analyzer_shared/PRESUBMIT.py 17.8s to run CheckChangeOnCommit from [...]/sdk/pkg/_fe_analyzer_shared/PRESUBMIT.py. Running [...]/sdk/pkg/front_end/PRESUBMIT.py Running [...]/sdk/pkg/frontend_server/PRESUBMIT.py Running [...]/sdk/pkg/kernel/PRESUBMIT.py Presubmit checks took 25.3s to calculate. Python 3 presubmit checks passed. real 0m26.585s user 1m8.997s sys 0m8.742s ``` Worth noting here is that "sdk/PRESUBMIT.py" takes 5+ seconds here Before: ``` [I2024-01-25 10:11:39,863 203026 140202046494592 presubmit_support.py] Found 8 file(s). Running Python 3 presubmit commit checks ... Running [...]/sdk/PRESUBMIT.py Running [...]/sdk/pkg/_fe_analyzer_shared/PRESUBMIT.py 14.6s to run CheckChangeOnCommit from [...]/sdk/pkg/_fe_analyzer_shared/PRESUBMIT.py. Running [...]/sdk/pkg/front_end/PRESUBMIT.py 28.0s to run CheckChangeOnCommit from [...]/sdk/pkg/front_end/PRESUBMIT.py. Running [...]/sdk/pkg/frontend_server/PRESUBMIT.py 20.9s to run CheckChangeOnCommit from [...]/sdk/pkg/frontend_server/PRESUBMIT.py. Running [...]/sdk/pkg/kernel/PRESUBMIT.py Presubmit checks took 75.6s to calculate. Python 3 presubmit checks passed. real 1m16.870s user 3m48.784s sys 0m23.689s ``` So from 76+ to ~27 seconds. In response to https://github.com/dart-lang/sdk/issues/54665 Change-Id: I59a43f5009bba8c2fdcb5d3a843b4cb408499214 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/348301 Commit-Queue: Jens Johansen <jensj@google.com> Reviewed-by: Johnni Winther <johnniwinther@google.com> |
||
|---|---|---|
| .. | ||
| bin | ||
| lib | ||
| test | ||
| analysis_options.yaml | ||
| OWNERS | ||
| PRESUBMIT.py | ||
| pubspec.yaml | ||
| README.md | ||
The frontend_server package is used by both the flutter command line tool and the frontend_server_client package (used by webdev and package:test).
API Stability
Changes to the command line API or behavior should be tested against the following test suites (in addition to normal HHH testing):
- flutter_tools: https://github.com/flutter/flutter/tree/master/packages/flutter_tools
- frontend_server_client: https://github.com/dart-lang/webdev/tree/master/frontend_server_client
Otherwise these changes will need to be carefully coordinated with the flutter tooling and frontend_server_client teams.
This API stability does not cover any of the source code APIs.
Stable subset
- The frontend_server kernel compilation and expression evaluation for kernel should be considered "stable".
- The frontend_server JavaScript compilation is semi-stable, but clients should anticipate coordinated breaking changes in the future.
- The frontend_server JavaScript expression evaluation is experimental and is expected to change significantly from Dec 2020 through the end of 2021.
- Specific flags may be added for experimentation and should not be considered stable.