Commit Graph

944 Commits

Author SHA1 Message Date
Victoria Ashworth
822ebc48e0
Allow all iOS tests to use either iOS 16 or 17 (#142714)
Undoing https://github.com/flutter/flutter/pull/142323 to allow all iOS tests to use iOS 16 or 17.
2024-02-02 16:02:08 +00:00
Ricardo Amador
c13ebf1e78
Use proto name for emulator version and show cipd package version (#142262)
Changes to use the proto.textpb config file as the version of the emulator device and show the cipd dependency version in the dependencies so devs know it can be updated.

*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Fixes https://github.com/flutter/flutter/issues/142261

*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
2024-02-01 23:48:19 +00:00
Flutter GitHub Bot
2c0cf448bd
Marks Linux_android_emu android views to be unflaky (#142590)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Linux_android_emu android views"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Linux_android_emu%20android%20views%22).
This test can be marked as unflaky.
2024-02-01 22:42:55 +00:00
Polina Cherkasova
05cd4fd042
Make leak_tracking bots sticked to the left even if bot thinks they are non-flacky. (#142744) 2024-02-01 21:47:54 +00:00
Flutter GitHub Bot
071830663a
Marks Mac_arm64_ios basic_material_app_ios__compile to be unflaky (#142594)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64_ios basic_material_app_ios__compile"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64_ios%20basic_material_app_ios__compile%22).
This test can be marked as unflaky.
2024-01-31 19:24:05 +00:00
Flutter GitHub Bot
9c107801ed
Marks Mac_arm64 tool_tests_commands to be unflaky (#142593)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 tool_tests_commands"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20tool_tests_commands%22).
This test can be marked as unflaky.
2024-01-31 18:55:25 +00:00
Flutter GitHub Bot
1d48b60d48
Marks Mac_x64 tool_tests_commands to be unflaky (#142592)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 tool_tests_commands"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20tool_tests_commands%22).
This test can be marked as unflaky.
2024-01-31 18:45:34 +00:00
Flutter GitHub Bot
0e274f5c7b
Marks Mac_x64_ios integration_test_test_ios to be unflaky (#142595)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64_ios integration_test_test_ios"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64_ios%20integration_test_test_ios%22).
This test can be marked as unflaky.
2024-01-31 18:42:28 +00:00
Flutter GitHub Bot
62fcc15887
Marks Mac_x64 native_ui_tests_macos to be unflaky (#142598)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 native_ui_tests_macos"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20native_ui_tests_macos%22).
This test can be marked as unflaky.
2024-01-31 18:34:01 +00:00
Flutter GitHub Bot
2dc7c92205
Marks Mac_x64_ios hot_mode_dev_cycle_ios__benchmark to be unflaky (#142597)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64_ios hot_mode_dev_cycle_ios__benchmark"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64_ios%20hot_mode_dev_cycle_ios__benchmark%22).
This test can be marked as unflaky.
2024-01-31 18:30:47 +00:00
Flutter GitHub Bot
5efcfba66c
Marks Mac_arm64 native_ui_tests_macos to be unflaky (#142599)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 native_ui_tests_macos"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20native_ui_tests_macos%22).
This test can be marked as unflaky.
2024-01-31 18:27:46 +00:00
Flutter GitHub Bot
8629dfa283
Marks Windows_android hot_mode_dev_cycle_win__benchmark to be flaky (#142609)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Windows_android hot_mode_dev_cycle_win__benchmark"
}
-->
Issue link: https://github.com/flutter/flutter/issues/142608
2024-01-31 18:27:44 +00:00
Flutter GitHub Bot
5f2961d2dd
Marks Mac_arm64_ios integration_test_test_ios to be unflaky (#142596)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64_ios integration_test_test_ios"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64_ios%20integration_test_test_ios%22).
This test can be marked as unflaky.
2024-01-31 18:01:22 +00:00
Flutter GitHub Bot
347403f2ea
Marks Linux_android_emu android_defines_test to be unflaky (#142591)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Linux_android_emu android_defines_test"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Linux_android_emu%20android_defines_test%22).
This test can be marked as unflaky.
2024-01-31 17:08:54 +00:00
Flutter GitHub Bot
4af2051f8e
Marks Windows_android native_assets_android to be unflaky (#140042)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Windows_android native_assets_android"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Windows_android%20native_assets_android%22).
This test can be marked as unflaky.
2024-01-31 04:13:30 +00:00
Flutter GitHub Bot
3a442ab4e5
Marks Linux_pixel_7pro complex_layout_scroll_perf_impeller_gles__timeline_summary to be unflaky (#140039)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Linux_pixel_7pro complex_layout_scroll_perf_impeller_gles__timeline_summary"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Linux_pixel_7pro%20complex_layout_scroll_perf_impeller_gles__timeline_summary%22).
This test can be marked as unflaky.
2024-01-31 04:01:29 +00:00
Flutter GitHub Bot
6379378c88
Marks Mac flavors_test_macos to be unflaky (#140647)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac flavors_test_macos"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac%20flavors_test_macos%22).
This test can be marked as unflaky.
2024-01-31 03:08:36 +00:00
Ian Hickson
cd4445345c
Be less verbose in CI for customer_testing, and revert old timeout hack (#139611)
This PR does two things. First, it reduces the verbosity in the customer_testing shard. We want to be less verbose because it's nigh on impossible to find anything when there's a failure in the verbose logs. Also, some additional comments are added to make following the breadcrumbs easier for the next person.

Second, it reverts a timeout that had been set to 90 minutes a while ago. It's no longer needed.

Fixes https://github.com/flutter/flutter/issues/120901.
2024-01-30 23:37:23 +00:00
Flutter GitHub Bot
bd715036b5
Marks Windows_arm64 plugin_test_windows to be unflaky (#137675)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Windows_arm64 plugin_test_windows"
}
-->
The issue https://github.com/flutter/flutter/issues/134083 has been closed, and the test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Windows_arm64%20plugin_test_windows%22).
This test can be marked as unflaky.
2024-01-30 18:02:52 +00:00
Flutter GitHub Bot
9207b85de8
Marks Mac_pixel_7pro native_assets_android to be unflaky (#141675)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_pixel_7pro native_assets_android"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_pixel_7pro%20native_assets_android%22).
This test can be marked as unflaky.
2024-01-29 22:16:06 +00:00
Flutter GitHub Bot
1af09dde12
Marks Mac_x64 framework_tests_misc to be unflaky (#142118)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 framework_tests_misc"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20framework_tests_misc%22).
This test can be marked as unflaky.
2024-01-29 21:02:25 +00:00
Flutter GitHub Bot
259c72a82d
Marks Mac_arm64 framework_tests_misc to be unflaky (#142119)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 framework_tests_misc"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20framework_tests_misc%22).
This test can be marked as unflaky.
2024-01-29 19:53:21 +00:00
Loïc Sharma
f727948f6d
[Windows Arm64] Run plugin test post-submit (#141987)
The Flutter try pool does not have any Windows Arm64 machines. Thus, any Windows Arm64 pre-submit tests will fail as there's no machines to run them.

This converts the only Windows Arm64 pre-submit test, the plugin test, to run post-submit.

See: https://github.com/flutter/flutter/issues/141986
2024-01-29 19:10:45 +00:00
Jenn Magder
aaa3eba12d
Limit fuchsia_precache in presubmit to engine rolls (#142333)
This doesn't need to run on every PR in presubmit.  Limit to engine rolls to validate the artifacts have been published.

Fixes https://github.com/flutter/flutter/issues/142330
2024-01-26 22:34:20 +00:00
Victoria Ashworth
e1785b315b
Only use iOS 17 physical devices in staging tests (#142323)
Don't use iOS 17 devices in pre and post submit tests yet.
2024-01-26 21:51:43 +00:00
Jenn Magder
91f0878fed
Move iOS content validation devicelab test into tool integration test (#142272)
The archiving was running in devicelab because certs are needed to codesign (see #73577).  However now the certs are available in chromium bots.  Move the archiving test into the existing tool integration test, and delete the devicelab variant.

arm64:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8757886514651624673/+/u/run_test.dart_for_tool_host_cross_arch_tests_shard_and_subshard_None/test_stdout#L6074_4
x64:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8757886514651624689/+/u/run_test.dart_for_tool_host_cross_arch_tests_shard_and_subshard_None/test_stdout#L6389_2

Part of https://github.com/flutter/flutter/issues/142070
2024-01-26 18:04:09 +00:00
Matan Lurey
2f6fdf2650
Start renaming by adding a new bringup: true as an Android emulator. (#142257)
Part 2 of https://github.com/flutter/flutter/issues/142178.

Supersedes https://github.com/flutter/flutter/pull/142078.

Feedback welcome to avoid reverts!
2024-01-26 01:07:08 +00:00
godofredoc
a6c3ad2310
Update name for android_defines_test. (#142273)
This is to reuse the global emulator configurations and make it easier to identify this test is running on an emulator.
2024-01-26 00:16:36 +00:00
godofredoc
8cc944d776
Run a few mac tests only on arm. (#142188)
Note: some benchmarks will require to be rebaselined.
2024-01-25 23:05:20 +00:00
godofredoc
3165fa30be
Migrate android_view to linux_android_emu platform. (#142184)
A new top level platform configuration was created for android emulator tests running on linux vms.

This is also a preparation step to remove adhoc recipes created to run emulator tests.
2024-01-25 01:15:12 +00:00
auto-submit[bot]
54e9f2dbe6
Reverts "Refactor external_uiexternal_textures" (#142173)
Reverts flutter/flutter#142062
Initiated by: eliasyishak
This change reverts the following previous change:
Original Description:
This PR makes no _behavioral_ changes to executed code, and instead focuses on organization and naming:

1. Almost[^1] anything named `external_ui` is renamed `external_textures`
1. Extended the README to explain the intent of the test, as well as how to run it
1. Renamed `main.dart` and `main_test.dart` to `frame_rate_main.dart` and `frame_rate_test.dart` (we'll add more)
1. Did some refactoring of the test to make it more obvious what is being asserted (i.e. `widgetBuilds` and friends)

Given how complex (and in-flux) this directory is, I'm also requesting either John, Jonah or I review any changes.

[^1]: Except the name of the `.ci.yaml` task, i.e. `name: Linux_pixel_7pro external_ui_integration_test` because I'm apparently not able to change that without creating a new task as `bringup: true` and playing a bit of a dance. Maybe that's worth doing though (in future PRs)?
2024-01-24 21:41:17 +00:00
Matan Lurey
2e2042ff8e
Refactor external_uiexternal_textures (#142062)
This PR makes no _behavioral_ changes to executed code, and instead
focuses on organization and naming:

1. Almost[^1] anything named `external_ui` is renamed
`external_textures`
1. Extended the README to explain the intent of the test, as well as how
to run it
1. Renamed `main.dart` and `main_test.dart` to `frame_rate_main.dart`
and `frame_rate_test.dart` (we'll add more)
1. Did some refactoring of the test to make it more obvious what is
being asserted (i.e. `widgetBuilds` and friends)

Given how complex (and in-flux) this directory is, I'm also requesting
either John, Jonah or I review any changes.

[^1]: Except the name of the `.ci.yaml` task, i.e. `name:
Linux_pixel_7pro external_ui_integration_test` because I'm apparently
not able to change that without creating a new task as `bringup: true`
and playing a bit of a dance. Maybe that's worth doing though (in future
PRs)?
2024-01-24 12:44:31 -08:00
Flutter GitHub Bot
a0df2c4573
Marks Mac_arm64 build_tests_2_4 to be unflaky (#142115)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 build_tests_2_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20build_tests_2_4%22).
This test can be marked as unflaky.
2024-01-24 18:11:53 +00:00
Flutter GitHub Bot
07ece1aad2
Marks Mac_x64 build_tests_2_4 to be unflaky (#142111)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 build_tests_2_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20build_tests_2_4%22).
This test can be marked as unflaky.
2024-01-24 17:56:38 +00:00
Flutter GitHub Bot
fdda3b5ac0
Marks Mac_x64 build_tests_1_4 to be unflaky (#142110)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 build_tests_1_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20build_tests_1_4%22).
This test can be marked as unflaky.
2024-01-24 17:56:35 +00:00
Flutter GitHub Bot
a0e0387a80
Marks Mac_x64 build_tests_3_4 to be unflaky (#142112)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 build_tests_3_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20build_tests_3_4%22).
This test can be marked as unflaky.
2024-01-24 17:55:43 +00:00
godofredoc
5d48b5c8b5
Run some tests explicitly in both arm and x64. (#141910)
This is part of the process to start running tests on either arm64 or x64 if the host architecture is not important for the test.
2024-01-24 17:24:12 +00:00
Flutter GitHub Bot
161fd82c7f
Marks Mac_arm64 build_tests_1_4 to be unflaky (#142114)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 build_tests_1_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20build_tests_1_4%22).
This test can be marked as unflaky.
2024-01-24 15:50:11 +00:00
Flutter GitHub Bot
277366d2cc
Marks Mac_arm64 build_tests_3_4 to be unflaky (#142116)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 build_tests_3_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20build_tests_3_4%22).
This test can be marked as unflaky.
2024-01-24 15:39:25 +00:00
Flutter GitHub Bot
db0ce8a498
Marks Mac_x64 build_tests_4_4 to be unflaky (#142113)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 build_tests_4_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_x64%20build_tests_4_4%22).
This test can be marked as unflaky.
2024-01-24 15:09:03 +00:00
Flutter GitHub Bot
e7ff0e0e8f
Marks Mac_arm64 build_tests_4_4 to be unflaky (#142117)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_arm64 build_tests_4_4"
}
-->
The test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Mac_arm64%20build_tests_4_4%22).
This test can be marked as unflaky.
2024-01-24 14:59:41 +00:00
godofredoc
6f7aed593d
Run module_test_ios on arm and x64 (#141815)
This will ensure the test runs on the supported platforms.
2024-01-19 22:05:12 +00:00
godofredoc
c95bb6fbb3
Add mac_x64_ios configuration. (#141828)
This is in preparation to allow tests to run on x64 and arm64 if they rely only on the ios phone.
2024-01-19 03:42:23 +00:00
godofredoc
9d9bd884ea
Run framework_tests_misc in arm64 and x64. (#141797)
This is to ensure coverage in both platforms.
2024-01-18 22:40:16 +00:00
Victoria Ashworth
c92b20ba97
Run flutter_gallery_ios__start_up test on Mac-14 in staging (#141795)
Attempting to debug https://github.com/flutter/flutter/issues/141383.
2024-01-18 20:05:10 +00:00
Zachary Anderson
a9d94be0da
Move mac pixel 7 pro test to presubmit: false (#141747)
The only bot with this configuration in the try pool just died.
2024-01-17 19:54:51 -08:00
Polina Cherkasova
b90a3e4710
Remove non-needed bot and increase time out for leak_tracking. (#141712) 2024-01-17 15:31:38 -08:00
godofredoc
66f5e88e6b
Run build tests on both x64 and arm64. (#141206)
Most of the builds for mac_x64 can actually run on any available architecture. This is required to use capacity more efficiently.
2024-01-17 21:20:34 +00:00
Victoria Ashworth
8de818dd2a
Update tests to Xcode 15 (#141706)
Framework part of https://github.com/flutter/flutter/issues/132237.
2024-01-17 20:32:12 +00:00
Flutter GitHub Bot
8286f4993d
Marks Windows framework_tests_misc_leak_tracking to be unflaky (#141676)
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Windows framework_tests_misc_leak_tracking"
}
-->
The issue https://github.com/flutter/flutter/issues/140414 has been closed, and the test has been passing for [50 consecutive runs](https://data.corp.google.com/sites/flutter_infra_metrics_datasite/flutter_check_test_flakiness_status_dashboard/?p=BUILDER_NAME:%22Windows%20framework_tests_misc_leak_tracking%22).
This test can be marked as unflaky.
2024-01-17 17:37:36 +00:00