This suite typically takes almost 30 minutes on CI, and some recent
engine->framework rolls failed because the tests exceeded the current 30
minute timeout.
On Linux hosts, there are currently 5 mokey's in staging and 7 in prod.
On Linux hosts, there is currently 1 MotoG4 in staging and 11 in prod.
This PR shifts the ~5 tests that have been running on mokey's for awhile
now from staging to prod, and shifts about half of the MotoG4 benchmarks
from MotoG4's in prod to mokey's in staging.
Following this change, we can disconnect half of the Linux/MotoG4's in
the prod pool and replace them with mokey's in the prod pool.
Part of https://github.com/flutter/flutter/issues/148085
We never made much use of the a02s's, so we can easily free up some
hosts for more mokey devices by eagerly shifting some of the tests on
a02s to mokey.
The test on MotoG4 has lots of flaky timeouts, so this PR moves it to
bringup to stop turning the tree red. This PR also adds the test on
Mokey in anticipation of removing the MotoG4 devices and to see if it
might be flaky there as well.
The macOS devicelab machines are not physically configured for benchmarking at a 60 Hz refresh rate. With Flutter vsync wired up via CVDisplayLink (rather than a hardcoded 60 Hz rate as previously), the bot shows 0 monitors configured and syncs at 30 Hz. This is not a meaningful target to measure against but consumes devicelab CPU cycles.
Given that there is currently no plan in place to acquire and configure a consistent pool of physical benchmarking desktop machines, disabling this test to reduce load on the bots.
See https://github.com/flutter/engine/pull/51210 for a discussion of the decision to move forward with wiring up vsync from macOS and ignore this benchmark.
For future archaeologists, I don't think it's worth removing all the plumbing in the Dart code (in [dev/devicelab/lib/tasks/perf_tests.dart](9d47055640/dev/devicelab/lib/tasks/perf_tests.dart (L963))) that supports running this benchmark on macOS. If we ever acquire hardware suitable for benchmarking with the appropriate HDMI dongle to simulate a 60Hz 4k display, for example, it would be reasonable to re-enable this benchmark.
Relands https://github.com/flutter/flutter/pull/146181.
Just 3 commits:
1. a revert of the revert
2. the fix described in https://github.com/flutter/flutter/pull/146181#issuecomment-2038238869
3. updating two postsubmit tests from Java 11 to 17, as that is required for this new AGP version.
I've verified that `flutter build apk --flavor paid --debug` fails in `dev/integration_tests/flavors/` with the error in ci without (2), and succeeds with it.
I've also verified that the `dev/benchmarks/complex_layout` app builds successfully with Java 17.
That covers all the postsubmits that failed according [to the dashboard](https://flutter-dashboard.appspot.com/#/build).
* Adds support for `flutter test --wasm`.
* The test compilation flow is a bit different now, so that it supports compilers other than DDC. Specifically, when we run a set of unit tests, we generate a "switchboard" main function that imports each unit test and runs the main function for a specific one based off of a value set by the JS bootstrapping code. This way, there is one compile step and the same compile output is invoked for each unit test file.
* Also, removes all references to `dart:html` from flutter/flutter.
* Adds CI steps for running the framework unit tests with dart2wasm+skwasm
* These steps are marked as `bringup: true`, so we don't know what kind of failures they will result in. Any failures they have will not block the tree at all yet while we're still in `bringup: true`. Once this PR is merged, I plan on looking at any failures and either fixing them or disabling them so we can get these CI steps running on presubmit.
This fixes https://github.com/flutter/flutter/issues/126692
Switch hot_mode_dev_cycle_linux__benchmark to run in postsubmit.
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Part of https://github.com/flutter/flutter/issues/145242
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
This is required to match a rename being made for the devices in firebase lab.
Also removed `version=24` entry, since thats not supported on Nexus5x, per the firebase team.
Bug:b/322830672
*Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.*
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Reverts: flutter/flutter#145224
Initiated by: hellohuanlin
Reason for reverting: breaks the tree
Original PR Author: hellohuanlin
Reviewed By: {gmackall, jmagman}
This change reverts the following previous change:
Reland https://github.com/flutter/flutter/pull/144745, which got reverted due to Android lockfile. Fixed by `dart dev/tools/bin/generate_gradle_lockfiles.dart`
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Fixes https://github.com/flutter/flutter/issues/143534
Fixes https://github.com/flutter/flutter/issues/143257
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
These tests were failing in postsubmit after a change to `packages/flutter_tools`. We should run these in presubmit when `packages/flutter_tools` changes.
Windows Defender sometimes kills the `Windows_arm64 plugin_test_windows` test process, causing the test to hang until it times out after 30 minutes. This reduces the test timeout to 900 seconds (15 minutes) to recover from this scenario faster.
Test timeout duration was chosen by looking at successful duration percentiles in the last 100 days:
duration_seconds_p90 | duration_seconds_p99 | duration_seconds_max
-- | -- | --
532 | 545 | 576
BigQuery SQL:
```sql
WITH
successful_steps AS (
SELECT
b.id,
TIMESTAMP_DIFF(s.end_time, s.start_time, SECOND) AS duration_seconds,
FROM cr-buildbucket.flutter.builds AS b, UNNEST(steps) AS s
WHERE
create_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 100 DAY)
AND regexp_substr(input.gitiles_commit.project, '[^\\/]+$') = 'flutter'
AND builder.project || '/' || builder.bucket || '/' || builder.builder
= 'flutter/prod/Windows_arm64 plugin_test_windows'
AND name = 'run plugin_test_windows'
AND s.status = 'SUCCESS'
)
SELECT
percentiles[offset(90)] AS duration_seconds_p90,
percentiles[offset(99)] AS duration_seconds_p99,
duration_seconds_max
FROM (
SELECT
APPROX_QUANTILES(duration_seconds, 100) AS percentiles,
MAX(duration_minutes) AS duration_seconds_max
FROM successful_steps
);
```
`test_timeout_secs` is documented here: https://github.com/flutter/cocoon/blob/main/CI_YAML.md
Part of https://github.com/flutter/flutter/issues/145072
Part of https://github.com/flutter/flutter/issues/143616
Without this these tests only run on postsubmit which means we can't get gold results on PRs.
This was reverted because it landed after an engine change that broke flutter tester. That change has since been reverted.
Almost all of our devicelab devices are on iOS 17 now, so let's use them exclusively so we can verify that tests pass consistently on iOS 17.
Fixes https://github.com/flutter/flutter/issues/144020.
