2016-09-14 20:22:53 +00:00
|
|
|
# Describes the tasks we run in the continuous integration (CI) environment.
|
|
|
|
#
|
|
|
|
# Cocoon[1] uses this file to generate a checklist of tasks to be performed for
|
|
|
|
# every master commit.
|
|
|
|
#
|
|
|
|
# [1] github.com/flutter/cocoon
|
|
|
|
|
|
|
|
# CI tasks.
|
|
|
|
#
|
|
|
|
# Each key in this dictionary is the unique name of a task, which also
|
|
|
|
# corresponds to a file in the "bin/" directory that the task runner will run.
|
|
|
|
#
|
2017-04-06 23:27:29 +00:00
|
|
|
# Required properties:
|
2016-09-14 20:22:53 +00:00
|
|
|
#
|
2017-04-06 23:27:29 +00:00
|
|
|
# * description: free form string
|
|
|
|
# describes what the task is about
|
|
|
|
# * stage: one of "devicelab", "devicelab_ios", "devicelab_win"
|
|
|
|
# tasks are grouped by stage so they appear next to each on the dashboard and have a distinct
|
|
|
|
# icon attached to them. Stages can be used to create dependencies, e.g. we may decide to not
|
2018-08-07 20:41:33 +00:00
|
|
|
# run device tests on red Cirrus builds.
|
2017-04-06 23:27:29 +00:00
|
|
|
# * required_agent_capabilities: a list of strings
|
|
|
|
# list of capabilities a devicelab agent must have to be able to run this task.
|
2016-09-14 20:22:53 +00:00
|
|
|
#
|
2017-04-06 23:27:29 +00:00
|
|
|
# Optional properties:
|
|
|
|
#
|
|
|
|
# * flaky: boolean true or false
|
|
|
|
# whether the task is considered flaky; the result of running a flaky task does not affect
|
2017-05-11 15:59:10 +00:00
|
|
|
# the overall build status. Tests should be marked flaky when newly added, until they have
|
|
|
|
# been proved for a few cycles. Tests that are actually flaky but not being actively worked
|
|
|
|
# on should be hidden because they just cause confusion otherwise.
|
2017-04-06 23:27:29 +00:00
|
|
|
# * timeout_in_minutes: integer
|
|
|
|
# a custom task timeout, specified in minutes.
|
2016-09-14 20:22:53 +00:00
|
|
|
|
|
|
|
tasks:
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
# Tests of compiling in a variety of modes
|
2016-09-14 20:22:53 +00:00
|
|
|
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
complex_layout_android__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the Complex
|
|
|
|
Layout sample app for Android from Linux.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
complex_layout_ios__compile:
|
2016-09-14 20:22:53 +00:00
|
|
|
description: >
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
Collects various performance metrics of compiling the Complex
|
|
|
|
Layout sample app for iOS from Mac.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
|
|
|
|
complex_layout_win__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the Complex
|
|
|
|
Layout for Android from Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
|
|
|
|
basic_material_app_android__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the default
|
|
|
|
app for Android from Linux.
|
2016-09-14 20:22:53 +00:00
|
|
|
stage: devicelab
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
|
|
|
basic_material_app_ios__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the default
|
|
|
|
app for iOS from Mac.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
|
|
|
|
basic_material_app_win__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the default
|
|
|
|
app for Android from Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2019-03-08 20:25:35 +00:00
|
|
|
codegen_integration_win:
|
2019-02-15 07:17:16 +00:00
|
|
|
description: >
|
|
|
|
Runs codegeneration and verifies that it can execute
|
|
|
|
correctly.
|
|
|
|
stage: devicelab_win
|
|
|
|
required_agent_capabilities: ["windows/android"]
|
|
|
|
|
2019-03-08 20:25:35 +00:00
|
|
|
codegen_integration_mac:
|
|
|
|
description: >
|
|
|
|
Runs codegeneration and verifies that it can execute
|
|
|
|
correctly.
|
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
|
|
|
|
codegen_integration_linux:
|
|
|
|
description: >
|
|
|
|
Runs codegeneration and verifies that it can execute
|
|
|
|
correctly.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2019-03-29 17:05:18 +00:00
|
|
|
uncaught_image_error_linux:
|
|
|
|
description: >
|
|
|
|
Ensures that an error thrown into the zone can be caught by the ImageStream
|
|
|
|
completer
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
flutter_gallery_android__compile:
|
2016-09-14 20:22:53 +00:00
|
|
|
description: >
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
Collects various performance metrics of compiling the Flutter
|
|
|
|
Gallery for Android from Linux.
|
2016-09-14 20:22:53 +00:00
|
|
|
stage: devicelab
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
|
|
|
flutter_gallery_ios__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the Flutter
|
|
|
|
Gallery for iOS from Mac.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
|
|
|
|
flutter_gallery_win__compile:
|
|
|
|
description: >
|
|
|
|
Collects various performance metrics of compiling the Flutter
|
|
|
|
Gallery for Android from Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2019-06-04 06:19:42 +00:00
|
|
|
windows_chrome_dev_mode:
|
|
|
|
description: >
|
|
|
|
Run flutter web on the devicelab and hot restart.
|
|
|
|
stage: devicelab_win
|
|
|
|
required_agent_capabilities: ["windows/android"]
|
2019-07-02 23:43:41 +00:00
|
|
|
flaky: true
|
2019-06-04 06:19:42 +00:00
|
|
|
|
2016-09-14 20:22:53 +00:00
|
|
|
# Android on-device tests
|
|
|
|
|
|
|
|
complex_layout_scroll_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of the Complex Layout sample app on
|
|
|
|
Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2018-08-02 23:47:58 +00:00
|
|
|
tiles_scroll_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of the tiles tab in the Complex Layout sample app on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2019-07-08 21:13:33 +00:00
|
|
|
tiles_scroll_perf_ios__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of the tiles tab in the Complex Layout sample app on iPhone 6.
|
|
|
|
stage: devicelab
|
|
|
|
flaky: true
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
|
|
|
|
tiles_scroll_perf_iphonexs__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of the tiles tab in the Complex Layout sample app on iPhone XS.
|
|
|
|
stage: devicelab
|
|
|
|
flaky: true
|
|
|
|
required_agent_capabilities: ["mac/iphonexs"]
|
|
|
|
|
2019-02-05 06:54:11 +00:00
|
|
|
home_scroll_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of scrolling the material page in the
|
|
|
|
flutter_gallery app on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2019-04-10 01:55:42 +00:00
|
|
|
drive_perf_debug_warning:
|
|
|
|
description: >
|
|
|
|
Check that driver will print warnings when traceAction is called in debug mode.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2018-12-18 06:01:07 +00:00
|
|
|
cull_opacity_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of culling opacity widgets on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2019-02-27 17:33:08 +00:00
|
|
|
cubic_bezier_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of cubic bezier animations on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2019-06-25 03:53:22 +00:00
|
|
|
backdrop_filter_perf__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of backdrop filter blurs on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
flaky: true
|
|
|
|
|
2017-08-23 08:55:35 +00:00
|
|
|
flavors_test:
|
|
|
|
description: >
|
|
|
|
Checks that flavored builds work on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-08-23 08:55:35 +00:00
|
|
|
|
2017-04-27 06:44:28 +00:00
|
|
|
channels_integration_test:
|
|
|
|
description: >
|
|
|
|
Checks that platform channels work on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-04-27 06:44:28 +00:00
|
|
|
|
2017-11-22 13:16:25 +00:00
|
|
|
external_ui_integration_test:
|
|
|
|
description: >
|
|
|
|
Checks that external UIs work on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-11-22 13:16:25 +00:00
|
|
|
|
2018-08-23 17:26:00 +00:00
|
|
|
embedded_android_views_integration_test:
|
|
|
|
description: >
|
|
|
|
Tests embedded Android views.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2018-09-06 19:31:18 +00:00
|
|
|
android_semantics_integration_test:
|
|
|
|
description: >
|
|
|
|
Tests that the Android accessibility bridge produces correct semantics.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2018-06-14 19:30:20 +00:00
|
|
|
run_release_test:
|
|
|
|
description: >
|
|
|
|
Checks that `flutter run --release` does not crash.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2017-08-11 08:38:12 +00:00
|
|
|
platform_interaction_test:
|
|
|
|
description: >
|
|
|
|
Checks platform interaction on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-08-11 08:38:12 +00:00
|
|
|
|
2017-04-27 06:44:28 +00:00
|
|
|
platform_channel_sample_test:
|
2017-03-20 21:19:03 +00:00
|
|
|
description: >
|
2017-03-27 09:17:31 +00:00
|
|
|
Runs a driver test on the Platform Channel sample app on Android.
|
2017-03-20 21:19:03 +00:00
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-03-20 21:19:03 +00:00
|
|
|
|
2018-05-16 11:24:47 +00:00
|
|
|
platform_view__start_up:
|
|
|
|
description: >
|
|
|
|
Verifies that Platform View can be used from an Android project.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2016-09-14 20:22:53 +00:00
|
|
|
complex_layout__start_up:
|
|
|
|
description: >
|
|
|
|
Measures the startup time of the Complex Layout sample app on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2016-09-19 17:03:26 +00:00
|
|
|
hot_mode_dev_cycle__benchmark:
|
|
|
|
description: >
|
|
|
|
Measures the performance of Dart VM hot patching feature.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2016-09-19 17:03:26 +00:00
|
|
|
|
2016-12-01 19:37:16 +00:00
|
|
|
complex_layout_scroll_perf__memory:
|
|
|
|
description: >
|
|
|
|
Measures memory usage of the scroll performance test.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2016-12-01 19:37:16 +00:00
|
|
|
|
2018-06-16 07:08:33 +00:00
|
|
|
hello_world_android__compile:
|
|
|
|
description: >
|
|
|
|
Measures the APK size of Hello World.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2016-12-03 00:50:27 +00:00
|
|
|
hello_world__memory:
|
|
|
|
description: >
|
|
|
|
Measures starting memory usage of the simplest Flutter app.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2016-12-03 00:50:27 +00:00
|
|
|
|
2017-01-09 22:57:14 +00:00
|
|
|
microbenchmarks:
|
|
|
|
description: >
|
|
|
|
Runs benchmarks from dev/benchmarks/microbenchmarks.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-04-06 17:48:45 +00:00
|
|
|
timeout_in_minutes: 30
|
2017-01-09 22:57:14 +00:00
|
|
|
|
2017-03-23 12:20:00 +00:00
|
|
|
flutter_view__start_up:
|
|
|
|
description: >
|
|
|
|
Verifies that Flutter View can be used from an Android project.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-03-23 12:20:00 +00:00
|
|
|
|
2017-05-11 21:29:20 +00:00
|
|
|
integration_ui:
|
|
|
|
description: >
|
|
|
|
Runs end-to-end Flutter tests on Android.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-05-11 21:29:20 +00:00
|
|
|
|
2017-07-19 19:57:22 +00:00
|
|
|
commands_test:
|
|
|
|
description: >
|
|
|
|
Runs tests of flutter run commands.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-07-19 19:57:22 +00:00
|
|
|
|
2018-02-07 16:58:23 +00:00
|
|
|
run_machine_concurrent_hot_reload:
|
|
|
|
description: >
|
|
|
|
Runs tests of concurrent hot reload commands via flutter run --machine.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
|
2017-08-31 18:38:21 +00:00
|
|
|
service_extensions_test:
|
|
|
|
description: >
|
|
|
|
Validates our service protocol extensions.
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-08-31 18:38:21 +00:00
|
|
|
|
2017-05-25 18:47:36 +00:00
|
|
|
android_sample_catalog_generator:
|
|
|
|
description: >
|
|
|
|
Builds sample catalog markdown pages and Android screenshots
|
|
|
|
stage: devicelab
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/android"]
|
2017-05-25 18:47:36 +00:00
|
|
|
|
2017-06-13 19:49:07 +00:00
|
|
|
complex_layout_semantics_perf:
|
|
|
|
description: >
|
|
|
|
Measures duration of building the initial semantics tree.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
Deep linking: automatically push the route hiearchy on load. (#10894)
The main purpose of this PR is to make it so that when you set the
initial route and it's a hierarchical route (e.g. `/a/b/c`), it
implies multiple pushes, one for each step of the route (so in that
case, `/`, `/a`, `/a/b`, and `/a/b/c`, in that order). If any of those
routes don't exist, it falls back to '/'.
As part of doing that, I:
* Changed the default for MaterialApp.initialRoute to honor the
actual initial route.
* Added a MaterialApp.onUnknownRoute for handling bad routes.
* Added a feature to flutter_driver that allows the host test script
and the device test app to communicate.
* Added a test to make sure `flutter drive --route` works.
(Hopefully that will also prove `flutter run --route` works, though
this isn't testing the `flutter` tool's side of that. My main
concern is over whether the engine side works.)
* Fixed `flutter drive` to output the right target file name.
* Changed how the stocks app represents its data, so that we can
show a page for a stock before we know if it exists.
* Made it possible to show a stock page that doesn't exist. It shows
a progress indicator if we're loading the data, or else shows a
message saying it doesn't exist.
* Changed the pathing structure of routes in stocks to work more
sanely.
* Made search in the stocks app actually work (before it only worked
if we happened to accidentally trigger a rebuild). Added a test.
* Replaced some custom code in the stocks app with a BackButton.
* Added a "color" feature to BackButton to support the stocks use case.
* Spaced out the ErrorWidget text a bit more.
* Added `RouteSettings.copyWith`, which I ended up not using.
* Improved the error messages around routing.
While I was in some files I made a few formatting fixes, fixed some
code health issues, and also removed `flaky: true` from some devicelab
tests that have been stable for a while. Also added some documentation
here and there.
2017-06-23 21:58:29 +00:00
|
|
|
|
2017-06-24 01:13:40 +00:00
|
|
|
routing_test:
|
Deep linking: automatically push the route hiearchy on load. (#10894)
The main purpose of this PR is to make it so that when you set the
initial route and it's a hierarchical route (e.g. `/a/b/c`), it
implies multiple pushes, one for each step of the route (so in that
case, `/`, `/a`, `/a/b`, and `/a/b/c`, in that order). If any of those
routes don't exist, it falls back to '/'.
As part of doing that, I:
* Changed the default for MaterialApp.initialRoute to honor the
actual initial route.
* Added a MaterialApp.onUnknownRoute for handling bad routes.
* Added a feature to flutter_driver that allows the host test script
and the device test app to communicate.
* Added a test to make sure `flutter drive --route` works.
(Hopefully that will also prove `flutter run --route` works, though
this isn't testing the `flutter` tool's side of that. My main
concern is over whether the engine side works.)
* Fixed `flutter drive` to output the right target file name.
* Changed how the stocks app represents its data, so that we can
show a page for a stock before we know if it exists.
* Made it possible to show a stock page that doesn't exist. It shows
a progress indicator if we're loading the data, or else shows a
message saying it doesn't exist.
* Changed the pathing structure of routes in stocks to work more
sanely.
* Made search in the stocks app actually work (before it only worked
if we happened to accidentally trigger a rebuild). Added a test.
* Replaced some custom code in the stocks app with a BackButton.
* Added a "color" feature to BackButton to support the stocks use case.
* Spaced out the ErrorWidget text a bit more.
* Added `RouteSettings.copyWith`, which I ended up not using.
* Improved the error messages around routing.
While I was in some files I made a few formatting fixes, fixed some
code health issues, and also removed `flaky: true` from some devicelab
tests that have been stable for a while. Also added some documentation
here and there.
2017-06-23 21:58:29 +00:00
|
|
|
description: >
|
|
|
|
Verifies that `flutter drive --route` still works. No performance numbers.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
2017-07-11 00:20:49 +00:00
|
|
|
|
|
|
|
flutter_gallery_instrumentation_test:
|
|
|
|
description: >
|
|
|
|
Same as flutter_gallery__transition_perf but uses Android instrumentation
|
|
|
|
framework, and therefore does not require a host computer to run. This
|
|
|
|
test can run on off-the-shelf infrastructures, such as Firebase Test Lab.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
2017-06-13 19:49:07 +00:00
|
|
|
|
2018-06-25 13:33:42 +00:00
|
|
|
flutter_attach_test:
|
|
|
|
description: >
|
|
|
|
Tests the `flutter attach` command.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2018-10-24 18:44:45 +00:00
|
|
|
named_isolates_test:
|
|
|
|
description: >
|
|
|
|
Tests naming and attaching to specific isolates.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2019-06-04 06:19:42 +00:00
|
|
|
linux_chrome_dev_mode:
|
|
|
|
description: >
|
|
|
|
Run flutter web on the devicelab and hot restart.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
flaky: true
|
|
|
|
|
2019-06-05 19:40:21 +00:00
|
|
|
web_size__compile_test:
|
|
|
|
description: >
|
|
|
|
Measures the size of a dart2js bundle.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2019-06-28 22:14:03 +00:00
|
|
|
image_list_reported_duration:
|
|
|
|
description: >
|
2019-07-08 15:23:06 +00:00
|
|
|
Measures image loading performance on release (aot) build.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
flaky: true
|
|
|
|
|
|
|
|
image_list_jit_reported_duration:
|
|
|
|
description: >
|
|
|
|
Measures image loading performance on debug (jit) build.
|
2019-06-28 22:14:03 +00:00
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
flaky: true
|
2019-06-05 19:40:21 +00:00
|
|
|
|
2019-07-03 17:54:04 +00:00
|
|
|
build_benchmark:
|
|
|
|
description: >
|
|
|
|
Measures APK build performance across config changes.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
flaky: true
|
|
|
|
|
2016-09-14 20:22:53 +00:00
|
|
|
# iOS on-device tests
|
|
|
|
|
2017-08-23 08:55:35 +00:00
|
|
|
flavors_test_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Checks that flavored builds work on iPhone 6.
|
2017-08-23 08:55:35 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-08-23 08:55:35 +00:00
|
|
|
|
2017-11-22 13:16:25 +00:00
|
|
|
external_ui_integration_test_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Checks that external UIs work on iPhone 6.
|
2017-11-22 14:14:14 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-11-22 13:16:25 +00:00
|
|
|
|
2017-04-27 06:44:28 +00:00
|
|
|
channels_integration_test_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Checks that platform channels work on iPhone 6.
|
2017-04-27 06:44:28 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-04-27 06:44:28 +00:00
|
|
|
|
2017-08-11 08:38:12 +00:00
|
|
|
platform_interaction_test_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Checks platform interaction on iPhone 6.
|
2017-08-11 08:38:12 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-08-11 08:38:12 +00:00
|
|
|
|
2017-04-27 06:44:28 +00:00
|
|
|
platform_channel_sample_test_ios:
|
2017-03-20 21:19:03 +00:00
|
|
|
description: >
|
2019-07-10 21:13:01 +00:00
|
|
|
Runs a driver test on the Platform Channel sample app on iPhone 6 Objective-C project.
|
2017-03-20 21:19:03 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-03-20 21:19:03 +00:00
|
|
|
|
2019-07-10 21:13:01 +00:00
|
|
|
platform_channel_sample_test_swift:
|
|
|
|
description: >
|
|
|
|
Runs a driver test on the Platform Channel sample app on iPhone 6 Swift project.
|
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
flaky: true
|
|
|
|
|
2018-05-16 11:24:47 +00:00
|
|
|
platform_view_ios__start_up:
|
|
|
|
description: >
|
|
|
|
Verifies that Platform View can be used from an iOS project.
|
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
|
2016-09-14 20:22:53 +00:00
|
|
|
complex_layout_scroll_perf_ios__timeline_summary:
|
|
|
|
description: >
|
|
|
|
Measures the runtime performance of the Complex Layout sample app on
|
|
|
|
iOS.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2017-11-02 21:20:14 +00:00
|
|
|
flutter_gallery_ios__start_up:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Measures the startup time of the Flutter Gallery app on iPhone 6.
|
2017-11-02 21:20:14 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2017-11-02 21:20:14 +00:00
|
|
|
complex_layout_ios__start_up:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Measures the startup time of the Complex Layout sample app on iPhone 6.
|
2017-11-02 21:20:14 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2016-09-14 20:22:53 +00:00
|
|
|
|
2017-03-30 02:19:57 +00:00
|
|
|
flutter_gallery_ios__transition_perf:
|
|
|
|
description: >
|
|
|
|
Measures the performance of screen transitions in Flutter Gallery on
|
|
|
|
iOS.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2016-12-05 20:46:59 +00:00
|
|
|
|
2018-06-16 07:08:33 +00:00
|
|
|
hello_world_ios__compile:
|
|
|
|
description: >
|
|
|
|
Measures the IPA size of Hello World.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
|
2017-11-02 21:20:14 +00:00
|
|
|
microbenchmarks_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Runs benchmarks from dev/benchmarks/microbenchmarks on iPhone 6.
|
2017-11-02 21:20:14 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-11-02 21:20:14 +00:00
|
|
|
timeout_in_minutes: 30
|
2017-05-11 15:59:10 +00:00
|
|
|
|
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-29 01:21:01 +00:00
|
|
|
flutter_view_ios__start_up:
|
|
|
|
description: >
|
|
|
|
Verifies that Flutter View can be used from an iOS project.
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-03-24 10:45:57 +00:00
|
|
|
|
2017-05-11 21:29:20 +00:00
|
|
|
integration_ui_ios:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Runs end-to-end Flutter tests on iPhone 6.
|
2017-05-11 21:29:20 +00:00
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2018-04-20 20:57:06 +00:00
|
|
|
timeout_in_minutes: 20
|
2017-05-11 21:29:20 +00:00
|
|
|
|
2017-05-25 18:47:36 +00:00
|
|
|
ios_sample_catalog_generator:
|
|
|
|
description: >
|
|
|
|
Builds sample catalog markdown pages and iOS screenshots
|
|
|
|
stage: devicelab_ios
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2017-05-25 18:47:36 +00:00
|
|
|
|
2019-06-04 06:19:42 +00:00
|
|
|
macos_chrome_dev_mode:
|
|
|
|
description: >
|
|
|
|
Run flutter web on the devicelab and hot restart.
|
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
2019-07-02 23:43:41 +00:00
|
|
|
flaky: true
|
2019-06-04 06:19:42 +00:00
|
|
|
|
2019-07-09 01:09:00 +00:00
|
|
|
build_benchmark_ios:
|
|
|
|
description: >
|
|
|
|
Measures iOS build performance across config changes.
|
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios"]
|
|
|
|
flaky: true
|
|
|
|
|
2017-03-16 20:47:14 +00:00
|
|
|
# Tests running on Windows host
|
|
|
|
|
2017-08-23 08:55:35 +00:00
|
|
|
flavors_test_win:
|
|
|
|
description: >
|
|
|
|
Checks that flavored builds work on Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
2017-08-23 08:55:35 +00:00
|
|
|
|
2017-04-27 06:44:28 +00:00
|
|
|
channels_integration_test_win:
|
|
|
|
description: >
|
|
|
|
Checks that platform channels work when app is launched from Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
2017-04-27 06:44:28 +00:00
|
|
|
|
2018-02-16 09:17:28 +00:00
|
|
|
plugin_test_win:
|
|
|
|
description: >
|
|
|
|
Checks that the project template works and supports plugins on Windows.
|
|
|
|
stage: devicelab_win
|
|
|
|
required_agent_capabilities: ["windows/android"]
|
|
|
|
|
2017-03-28 21:22:03 +00:00
|
|
|
hot_mode_dev_cycle_win__benchmark:
|
|
|
|
description: >
|
|
|
|
Measures the performance of Dart VM hot patching feature on Windows.
|
|
|
|
stage: devicelab_win
|
2018-01-03 03:53:11 +00:00
|
|
|
required_agent_capabilities: ["windows/android"]
|
2017-05-31 19:43:18 +00:00
|
|
|
|
2019-03-22 21:32:36 +00:00
|
|
|
run_without_leak_win:
|
|
|
|
description: >
|
|
|
|
Checks that `flutter run` does not leak dart.exe on Windows.
|
|
|
|
stage: devicelab_win
|
|
|
|
required_agent_capabilities: ["windows/android"]
|
|
|
|
|
2017-05-31 19:43:18 +00:00
|
|
|
# Tests running on Linux hosts
|
|
|
|
|
|
|
|
hot_mode_dev_cycle_linux__benchmark:
|
|
|
|
description: >
|
|
|
|
Measures the performance of Dart VM hot patching feature on a Linux host.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2018-06-27 00:34:25 +00:00
|
|
|
flutter_test_performance:
|
|
|
|
description: >
|
|
|
|
Measures performance of running flutter test.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2017-05-31 19:43:18 +00:00
|
|
|
technical_debt__cost:
|
|
|
|
description: >
|
|
|
|
Estimates our technical debt (TODOs, analyzer ignores, etc).
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
2017-05-31 22:13:08 +00:00
|
|
|
|
|
|
|
flutter_gallery__start_up:
|
|
|
|
description: >
|
|
|
|
Measures the startup time of the Flutter Gallery app on Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
|
|
|
flutter_gallery__transition_perf:
|
|
|
|
description: >
|
|
|
|
Measures the performance of screen transitions in Flutter Gallery on
|
|
|
|
Android.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2017-06-14 15:09:05 +00:00
|
|
|
flutter_gallery__transition_perf_with_semantics:
|
|
|
|
description: >
|
|
|
|
Measures the delta in performance of screen transitions without and
|
|
|
|
with semantics enabled.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2017-05-31 22:13:08 +00:00
|
|
|
flutter_gallery__memory_nav:
|
|
|
|
description: >
|
|
|
|
Measures memory usage after repeated navigation in Gallery.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
|
|
|
flutter_gallery__back_button_memory:
|
|
|
|
description: >
|
|
|
|
Measures memory usage after Android app suspend and resume.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
2019-06-13 06:06:20 +00:00
|
|
|
flaky: true
|
2017-06-13 20:14:16 +00:00
|
|
|
|
2019-06-04 19:32:52 +00:00
|
|
|
flutter_gallery__image_cache_memory:
|
|
|
|
description: >
|
|
|
|
Measures memory usage for a list of large red squares in smaller containers.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
|
2017-06-13 20:14:16 +00:00
|
|
|
analyzer_benchmark:
|
|
|
|
description: >
|
|
|
|
Measures the speed of Dart analyzer.
|
|
|
|
stage: devicelab
|
2018-08-13 20:26:32 +00:00
|
|
|
required_agent_capabilities: ["linux/android"]
|
2018-05-14 22:05:38 +00:00
|
|
|
|
2019-03-22 21:32:36 +00:00
|
|
|
run_without_leak_linux:
|
|
|
|
description: >
|
|
|
|
Checks that `flutter run` does not leak dart on Linux.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["linux/android"]
|
|
|
|
flaky: true
|
|
|
|
|
2018-05-16 16:29:10 +00:00
|
|
|
flutter_gallery_ios32__start_up:
|
|
|
|
description: >
|
2019-07-08 21:13:33 +00:00
|
|
|
Measures the startup time of the Flutter Gallery app on 32-bit iOS (iPhone 4S).
|
2018-05-16 16:29:10 +00:00
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios32"]
|
|
|
|
|
2018-05-14 22:05:38 +00:00
|
|
|
flutter_gallery_ios32__transition_perf:
|
|
|
|
description: >
|
|
|
|
Measures the performance of screen transitions in Flutter Gallery on
|
2019-07-08 21:13:33 +00:00
|
|
|
32-bit iOS (iPhone 4S).
|
2018-05-14 22:05:38 +00:00
|
|
|
stage: devicelab_ios
|
|
|
|
required_agent_capabilities: ["mac/ios32"]
|
2019-03-22 21:32:36 +00:00
|
|
|
|
|
|
|
run_without_leak_mac:
|
|
|
|
description: >
|
|
|
|
Checks that `flutter run` does not leak dart on macOS.
|
|
|
|
stage: devicelab
|
|
|
|
required_agent_capabilities: ["mac/android"]
|
|
|
|
flaky: true
|