2020-10-23 13:37:03 +00:00
|
|
|
# Flutter DeviceLab
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
DeviceLab is a physical lab that tests Flutter on real devices.
|
2016-10-06 17:20:58 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
This package contains the code for the test framework and tests. More generally
|
2016-10-06 17:20:58 +00:00
|
|
|
the tests are referred to as "tasks" in the API, but since we primarily use it
|
|
|
|
for testing, this document refers to them as "tests".
|
|
|
|
|
2020-02-12 02:25:17 +00:00
|
|
|
Current statuses for the devicelab are available at
|
2022-12-16 23:14:18 +00:00
|
|
|
<https://flutter-dashboard.appspot.com/#/build>. See [dashboard user
|
2023-12-29 05:45:36 +00:00
|
|
|
guide](https://github.com/flutter/cocoon/blob/main/dashboard/USER_GUIDE.md)
|
2020-10-23 13:37:03 +00:00
|
|
|
for information on using the dashboards.
|
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
## Table of Contents
|
2022-12-16 23:14:18 +00:00
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
* [How the DeviceLab runs tests](#how-the-devicelab-runs-tests)
|
|
|
|
* [Running tests locally](#running-tests-locally)
|
|
|
|
* [Writing tests](#writing-tests)
|
2022-12-16 23:14:18 +00:00
|
|
|
* [Adding tests to continuous
|
|
|
|
integration](#adding-tests-to-continuous-integration)
|
2021-04-14 20:49:02 +00:00
|
|
|
* [Adding tests to presubmit](#adding-tests-to-presubmit)
|
2023-07-27 15:16:09 +00:00
|
|
|
* [Migrating to build and test model](#migrating-to-build-and-test-model)
|
2020-10-23 13:37:03 +00:00
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
## How the DeviceLab runs tests
|
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
DeviceLab tests are run against physical devices in Flutter's lab (the
|
|
|
|
"DeviceLab").
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`,
|
|
|
|
`mac_android`, `windows_android`, etc). When a device in the lab is free, it
|
|
|
|
will pickup tasks that need to be completed.
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
1. If the task succeeds, the test runner reports the success and uploads its
|
|
|
|
performance metrics to Flutter's infrastructure. Not all tasks record
|
|
|
|
performance metrics.
|
|
|
|
2. If task fails, an auto rerun happens. Whenever the last run succeeds, the
|
|
|
|
task will be reported as a success. For this case, a flake will be flagged and
|
|
|
|
populated to the test result.
|
|
|
|
3. If the task fails in all reruns, the test runner reports the failure to
|
|
|
|
Flutter's infrastructure and no performance metrics are collected
|
2016-10-06 17:20:58 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
## Running tests locally
|
2016-10-06 17:20:58 +00:00
|
|
|
|
|
|
|
Do make sure your tests pass locally before deploying to the CI environment.
|
|
|
|
Below is a handful of commands that run tests in a similar way to how the
|
|
|
|
CI environment runs them. These commands are also useful when you need to
|
|
|
|
reproduce a CI test failure locally.
|
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
### Prerequisites
|
2018-02-13 00:51:25 +00:00
|
|
|
|
2020-07-23 01:23:47 +00:00
|
|
|
You must set the `ANDROID_SDK_ROOT` environment variable to run
|
2018-12-17 17:29:09 +00:00
|
|
|
tests on Android. If you have a local build of the Flutter engine, then you have
|
|
|
|
a copy of the Android SDK at `.../engine/src/third_party/android_tools/sdk`.
|
2018-02-13 00:51:25 +00:00
|
|
|
|
2022-12-12 22:39:49 +00:00
|
|
|
You can find where your Android SDK is using `flutter doctor -v`.
|
2018-07-30 23:58:07 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
### Warnings
|
2018-08-28 18:28:25 +00:00
|
|
|
|
2020-07-23 01:23:47 +00:00
|
|
|
Running the devicelab will do things to your environment.
|
2018-08-28 18:28:25 +00:00
|
|
|
|
2020-07-23 01:23:47 +00:00
|
|
|
Notably, it will start and stop Gradle, for instance.
|
2018-08-28 18:28:25 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
### Running specific tests
|
2018-07-30 23:58:07 +00:00
|
|
|
|
2016-10-06 17:20:58 +00:00
|
|
|
To run a test, use option `-t` (`--task`):
|
|
|
|
|
|
|
|
```sh
|
2016-11-15 20:35:18 +00:00
|
|
|
# from the .../flutter/dev/devicelab directory
|
2023-11-29 08:41:27 +00:00
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/test_runner.dart test -t {NAME_OF_TEST}
|
2016-10-06 17:20:58 +00:00
|
|
|
```
|
|
|
|
|
2023-11-29 08:41:27 +00:00
|
|
|
Where `NAME_OR_PATH_OF_TEST` is the name of a task, which is a file's
|
|
|
|
basename in `bin/tasks`. Example: `complex_layout__start_up`.
|
2016-11-15 20:35:18 +00:00
|
|
|
|
2016-10-06 17:20:58 +00:00
|
|
|
To run multiple tests, repeat option `-t` (`--task`) multiple times:
|
|
|
|
|
|
|
|
```sh
|
2019-01-19 08:31:05 +00:00
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t test1 -t test2 -t test3
|
2016-10-06 17:20:58 +00:00
|
|
|
```
|
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
### Running tests against a local engine build
|
2019-07-24 20:44:13 +00:00
|
|
|
|
|
|
|
To run device lab tests against a local engine build, pass the appropriate
|
|
|
|
flags to `bin/run.dart`:
|
|
|
|
|
|
|
|
```sh
|
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/run.dart --task=[some_task] \
|
|
|
|
--local-engine-src-path=[path_to_local]/engine/src \
|
2023-08-11 00:25:30 +00:00
|
|
|
--local-engine=[local_engine_architecture] \
|
|
|
|
--local-engine-host=[local_engine_host_architecture]
|
2019-07-24 20:44:13 +00:00
|
|
|
```
|
|
|
|
|
2023-08-11 00:25:30 +00:00
|
|
|
An example of a local engine architecture is `android_debug_unopt_x86` and
|
|
|
|
an example of a local engine host architecture is `host_debug_unopt`.
|
2019-07-24 20:44:13 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
### Running an A/B test for engine changes
|
2020-04-10 23:53:52 +00:00
|
|
|
|
|
|
|
You can run an A/B test that compares the performance of the default engine
|
|
|
|
against a local engine build. The test runs the same benchmark a specified
|
|
|
|
number of times against both engines, then outputs a tab-separated spreadsheet
|
2020-05-08 07:34:02 +00:00
|
|
|
with the results and stores them in a JSON file for future reference. The
|
|
|
|
results can be copied to a Google Spreadsheet for further inspection and the
|
2020-07-23 01:23:47 +00:00
|
|
|
JSON file can be reprocessed with the `summarize.dart` command for more detailed
|
2020-05-08 07:34:02 +00:00
|
|
|
output.
|
2020-04-10 23:53:52 +00:00
|
|
|
|
|
|
|
Example:
|
|
|
|
|
|
|
|
```sh
|
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/run.dart --ab=10 \
|
|
|
|
--local-engine=host_debug_unopt \
|
2023-08-11 00:25:30 +00:00
|
|
|
--local-engine-host=host_debug_unopt \
|
2020-04-10 23:53:52 +00:00
|
|
|
-t bin/tasks/web_benchmarks_canvaskit.dart
|
|
|
|
```
|
|
|
|
|
|
|
|
The `--ab=10` tells the runner to run an A/B test 10 times.
|
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
`--local-engine=host_debug_unopt` tells the A/B test to use the
|
2023-08-11 00:25:30 +00:00
|
|
|
`host_debug_unopt` engine build. `--local-engine-host=host_debug_unopt` uses
|
|
|
|
the same engine build to run the `frontend_server` (in this example).
|
|
|
|
`--local-engine` is required for A/B test.
|
2020-04-10 23:53:52 +00:00
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
`--ab-result-file=filename` can be used to provide an alternate location to
|
|
|
|
output the JSON results file (defaults to `ABresults#.json`). A single `#`
|
|
|
|
character can be used to indicate where to insert a serial number if a file with
|
|
|
|
that name already exists, otherwise, the file will be overwritten.
|
2020-05-08 07:34:02 +00:00
|
|
|
|
2020-04-10 23:53:52 +00:00
|
|
|
A/B can run exactly one task. Multiple tasks are not supported.
|
|
|
|
|
|
|
|
Example output:
|
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
```text
|
2020-04-10 23:53:52 +00:00
|
|
|
Score Average A (noise) Average B (noise) Speed-up
|
|
|
|
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
|
|
|
|
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
|
|
|
|
draw_rect.canvaskit.windowRenderDuration.average 1959.45 (16.56%) 2286.65 (0.61%) 0.86x
|
|
|
|
draw_rect.canvaskit.sceneBuildDuration.average 1969.45 (16.37%) 2294.90 (0.58%) 0.86x
|
|
|
|
draw_rect.canvaskit.drawFrameDuration.average 5335.20 (17.59%) 6437.60 (0.59%) 0.83x
|
|
|
|
draw_rect.canvaskit.totalUiFrame.average 6832.00 (13.16%) 7932.00 (0.34%) 0.86x
|
|
|
|
```
|
|
|
|
|
|
|
|
The output contains averages and noises for each score. More importantly, it
|
|
|
|
contains the speed-up value, i.e. how much _faster_ is the local engine than
|
|
|
|
the default engine. Values less than 1.0 indicate a slow-down. For example,
|
|
|
|
0.5x means the local engine is twice as slow as the default engine, and 2.0x
|
|
|
|
means it's twice as fast. Higher is better.
|
|
|
|
|
2020-05-08 07:34:02 +00:00
|
|
|
Summarize tool example:
|
|
|
|
|
|
|
|
```sh
|
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/summarize.dart --[no-]tsv-table --[no-]raw-summary \
|
|
|
|
ABresults.json ABresults1.json ABresults2.json ...
|
|
|
|
```
|
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
`--[no-]tsv-table` tells the tool to print the summary in a table with tabs for
|
|
|
|
easy spreadsheet entry. (defaults to on)
|
2020-05-08 07:34:02 +00:00
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
`--[no-]raw-summary` tells the tool to print all per-run data collected by the
|
|
|
|
A/B test formatted with tabs for easy spreadsheet entry. (defaults to on)
|
2020-05-08 07:34:02 +00:00
|
|
|
|
2022-12-16 23:14:18 +00:00
|
|
|
Multiple trailing filenames can be specified and each such results file will be
|
|
|
|
processed in turn.
|
2020-05-08 07:34:02 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
## Reproducing broken builds locally
|
2016-10-06 17:20:58 +00:00
|
|
|
|
|
|
|
To reproduce the breakage locally `git checkout` the corresponding Flutter
|
|
|
|
revision. Note the name of the test that failed. In the example above the
|
|
|
|
failing test is `flutter_gallery__transition_perf`. This name can be passed to
|
|
|
|
the `run.dart` command. For example:
|
|
|
|
|
|
|
|
```sh
|
2019-01-19 08:31:05 +00:00
|
|
|
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t flutter_gallery__transition_perf
|
2016-10-06 17:20:58 +00:00
|
|
|
```
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2020-10-23 13:37:03 +00:00
|
|
|
## Writing tests
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2019-08-28 23:59:45 +00:00
|
|
|
A test is a simple Dart program that lives under `bin/tasks` and uses
|
2016-09-14 20:22:53 +00:00
|
|
|
`package:flutter_devicelab/framework/framework.dart` to define and run a _task_.
|
|
|
|
|
|
|
|
Example:
|
|
|
|
|
|
|
|
```dart
|
|
|
|
import 'dart:async';
|
|
|
|
|
|
|
|
import 'package:flutter_devicelab/framework/framework.dart';
|
|
|
|
|
2018-10-04 16:44:23 +00:00
|
|
|
Future<void> main() async {
|
2016-09-14 20:22:53 +00:00
|
|
|
await task(() async {
|
|
|
|
... do something interesting ...
|
|
|
|
|
|
|
|
// Aggregate results into a JSONable Map structure.
|
|
|
|
Map<String, dynamic> testResults = ...;
|
|
|
|
|
|
|
|
// Report success.
|
|
|
|
return new TaskResult.success(testResults);
|
|
|
|
|
|
|
|
// Or you can also report a failure.
|
|
|
|
return new TaskResult.failure('Something went wrong!');
|
|
|
|
});
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
Only one `task` is permitted per program. However, that task can run any number
|
|
|
|
of tests internally. A task has a name. It succeeds and fails independently of
|
|
|
|
other tasks, and is reported to the dashboard independently of other tasks.
|
|
|
|
|
|
|
|
A task runs in its own standalone Dart VM and reports results via Dart VM
|
|
|
|
service protocol. This ensures that tasks do not interfere with each other and
|
|
|
|
lets the CI system time out and clean up tasks that get stuck.
|
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
## Adding tests to continuous integration
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
Host only tests should be added to `flutter_tools`.
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
There are several PRs needed to add a DeviceLab task to CI.
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2021-04-14 20:49:02 +00:00
|
|
|
_TASK_- the name of your test that also matches the name of the
|
2020-07-23 01:23:47 +00:00
|
|
|
file in `bin/tasks` without the `.dart` extension.
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2021-12-01 01:29:06 +00:00
|
|
|
1. Add target to
|
2024-04-17 13:16:33 +00:00
|
|
|
[.ci.yaml](https://github.com/flutter/flutter/blob/main/.ci.yaml)
|
2022-12-16 23:14:18 +00:00
|
|
|
* Mirror an existing one that has the recipe `devicelab_drone`
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2021-12-01 01:29:06 +00:00
|
|
|
If your test needs to run on multiple operating systems, create a separate
|
|
|
|
target for each operating system.
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2021-12-01 01:29:06 +00:00
|
|
|
## Adding tests to presubmit
|
2021-04-14 20:49:02 +00:00
|
|
|
|
2021-12-01 01:29:06 +00:00
|
|
|
Flutter's DeviceLab has a limited capacity in presubmit. File an infra ticket
|
|
|
|
to investigate feasibility of adding a test to presubmit.
|
2023-07-27 15:16:09 +00:00
|
|
|
|
|
|
|
## Migrating to build and test model
|
|
|
|
|
|
|
|
To better utilize limited DeviceLab testbed resources and speed up commit validation
|
|
|
|
time, it is now supported to separate building artifacts (.apk/.app) from testing them.
|
|
|
|
The artifact will be built on a host only bot, a VM or physical bot without a device,
|
|
|
|
and the test will run based on the artifact against a testbed with a device.
|
|
|
|
|
|
|
|
Steps:
|
|
|
|
|
2024-04-17 13:16:33 +00:00
|
|
|
1. Update the task class to extend [`BuildTestTask`](https://github.com/flutter/flutter/blob/main/dev/devicelab/lib/tasks/build_test_task.dart)
|
2023-07-27 15:16:09 +00:00
|
|
|
- Override function `getBuildArgs`
|
|
|
|
- Override function `getTestArgs`
|
|
|
|
- Override function `parseTaskResult`
|
|
|
|
- Override function `getApplicationBinaryPath`
|
|
|
|
2. Update the `bin/tasks/{TEST}.dart` to point to the new task class
|
|
|
|
3. Validate the task locally
|
|
|
|
- build only: `dart bin/test_runner.dart test -t {NAME_OR_PATH_OF_TEST} --task-args build --task-args application-binary-path={PATH_TO_ARTIFACT}`
|
|
|
|
- test only: `dart bin/test_runner.dart test -t {NAME_OR_PATH_OF_TEST} --task-args test --task-args application-binary-path={PATH_TO_ARTIFACT}`
|
|
|
|
4. Add tasks to continuous integration
|
|
|
|
- Mirror a target with platform `Linux_build_test` or `Mac_build_test`
|
|
|
|
- The only difference from regular targets is the artifact property: if omitted, it will use the `task_name`.
|
|
|
|
5. Once validated in CI, enable the target in `PROD` by removing `bringup: true` and deleting the old target entry without build+test model.
|
|
|
|
|
|
|
|
Take gallery tasks for example:
|
|
|
|
|
|
|
|
1. Linux android
|
|
|
|
- Separating PR: https://github.com/flutter/flutter/pull/103550
|
|
|
|
- Switching PR: https://github.com/flutter/flutter/pull/110533
|
2023-11-29 08:41:27 +00:00
|
|
|
2. Mac iOS: https://github.com/flutter/flutter/pull/111164
|