Commit graph

164 commits

Author SHA1 Message Date
Dan Field 2a644f301e
Run non-perf sensitive tests on Cirrus (#27971)
* Run non-perf sensisitive tests on Cirrus
2019-03-10 07:52:44 -07:00
Jonah Williams d4a441a0d9
Add integration to all targets (#29020) 2019-03-08 12:25:35 -08:00
liyuqian 19a6a6f431
Reland "Shader warm up (#27660)" (#28537)
This reverts commit adc8e159a5.

This should be safe to land once https://github.com/flutter/flutter/pull/28530 gets merged

Merge on yellow doc test because the doc test is actually green.
2019-02-27 09:33:08 -08:00
liyuqian 2e3d9ea253
Mark non-flaky test as such (#28373) 2019-02-25 10:19:49 -08:00
liyuqian adc8e159a5
Revert "Shader warm up (#27660)" (#28376)
This reverts commit a44f174efc.

Reason: start_up tests become flaky.

See https://github.com/flutter/flutter/issues/28374

TBR: xster

Merge on red to fix the tree
2019-02-22 17:32:37 -08:00
liyuqian a44f174efc
Shader warm up (#27660)
This patch adds a default shader warm up process which moves shader compilation from the animation time to the startup time. This also provides an extension for `runApp` so developers can customize the warm up process.

This should reduce our worst_frame_rasterizer_time_millis from ~100ms to ~20-30ms for both flutter_gallery and complex_layout benchmarks. Besides, this should also have a significant improvement on 90th and 99th percentile time (50%-100% speedup in some cases, but I haven't tested them thoroughly; I'll let our device lab collect the data afterwards).

The tradeoff the is the startup time (time to first frame). Our `flutter run --profile --trace-startup` seems to be a little noisy and I see about 100ms-200ms increase in that measurement for complex_layout and flutter_gallery. Note that this only happens on the first run after install or data wipe. Later the Skia persistent cache will remove the overhead.

This also adds a cubic_bezier benchmark to test the custom shader warm up process.

This should fix https://github.com/flutter/flutter/issues/813 (either by `defaultShaderWarmUp`, or a `customShaderWarmUp`).
2019-02-22 15:37:02 -08:00
Jonah Williams 67cf21577f
Add basic codegen app to be used for integration testing and benchmarks (#27257) 2019-02-14 23:17:16 -08:00
liyuqian de321b9cc1
Add scroll performance test for flutter_gallery (#27513)
Previously, we had a scroll_perf drive test but we were not exercising
it on our device lab. This helps us tracking issues like:
https://github.com/flutter/flutter/issues/24782
2019-02-04 22:54:11 -08:00
Dan Field ff170ab5f3
mark tests as not flaky (#26334) 2019-01-09 18:55:42 -08:00
Amir Hardon ff9f7236e6
Mark flutter_gallery__back_button_memory as flaky. (#25857)
See https://github.com/flutter/flutter/issues/25855
2018-12-28 15:34:15 -08:00
liyuqian 8ad4cbe06e
Add cull opacity perf test to device lab (#25381)
For https://github.com/flutter/flutter/issues/24712

This test verifies that https://github.com/flutter/engine/pull/6923 will speedup the average rasterize time of this test from ~150ms to ~10ms

Please see non-auto-generated files in 37b21d9fb4
2018-12-17 22:01:07 -08:00
liyuqian 7ecef3aa27
Mark flaky tests as such (#25482)
See https://github.com/flutter/flutter/issues/25481
2018-12-17 15:56:26 -08:00
Jonah Williams d1deb32fe9
try disabling flutter run test (#25416) 2018-12-15 10:54:47 -08:00
Jonah Williams f30029bae1
Run flutter tests through mini test engine when run directly (flutter run -t test_file) (#24930) 2018-12-12 12:51:16 -08:00
Dan Field 2729385758
Make host app test actually test host app (#24131)
* Make host app test actually test host app

* mark updated test as flaky
2018-11-09 19:09:56 -08:00
Dan Field f1f4bda8da
Mark some tests as not flaky (#23840)
These tests have been run for a few days now with no flakiness.
2018-11-01 20:08:45 -07:00
Yegor 223826247f
deflake the dartdocs test (#23582) 2018-10-26 13:12:14 -07:00
Yegor 4452bd0a6b
Revert "Revert "make LocalHistoryRoute a proper super-mixin (#23382)"… (#23482)
* Revert "Revert "make LocalHistoryRoute a proper super-mixin (#23382)" (#23430)"

This reverts commit 3bbb3082b8.

This relands the LocalHistoryRoute change. The analyzer bug has been fixed.

* mark dartdocs as flaky
2018-10-25 09:56:23 -07:00
Michael Klimushyn ce434022b5
Fix "E2E test setting and using isolate names (#23388)" (#23438)
Updates gradle.

This reverts commit d07c09c9a1.
2018-10-24 11:44:45 -07:00
Michael Goderbauer 55fe15bfdd
mark integration_ui_ios as flaky (#23437) 2018-10-23 16:45:17 -07:00
Michael Klimushyn e70f19523b
Revert "E2E test setting and using isolate names (#23388)" (#23416)
This reverts commit 93573de216.

Fails in the devicelab:

```
run:stderr: A problem occurred evaluating project ':app'.
run:stderr: > Could not resolve all files for configuration 'classpath'.2018-10-23T10:46:46.335864:
run:stderr:    > Could not find aapt2-proto.jar (com.android.tools.build:aapt2-proto:0.3.1).2018-10-23T10:46:46.335960:
run:stderr:      Searched in the following locations:2018-10-23T10:46:46.336048:
```
2018-10-23 11:28:12 -07:00
Michael Klimushyn 93573de216
E2E test setting and using isolate names (#23388)
Adds an integration devicelab test that runs an Android app with two
custom named isolates. Tests that the isolate names are present and that
it's possible to attach to just one of the isolates.

Fixes flutter/flutter#22009
2018-10-23 09:30:00 -07:00
Michael Klimushyn 43c99404c5
Test flutter create --offline (#23278)
Verifies that flutter tools isn't rebuilt. Doesn't truly check that the
created output is runnable.
2018-10-22 16:39:28 -07:00
Greg Spencer 0ff9e8a928
Rename 'application' back to 'module', and make 'app' the default again for templates. (#22888)
We decided that redefining the default for templates was premature. We're going to go back to having "module" in experimental land again, and we'll try again when we have the feature set fully baked.

This keeps the writing of the .metadata files, and writing the template type to them, because that was a good improvement, and there are still a bunch of added tests that improve our coverage.
2018-10-10 11:01:40 -07:00
Greg Spencer abb1758edf
Fix places in devicelab tests where it expects the app template. (#22696)
I missed some create template conversions in the devicelab directory.
2018-10-04 15:30:08 -07:00
Danny Tuppeny 55d70506ed
Mark module_test_ios as not-flaky (#22626)
The only reds on the dashboard for this test are on rows that have many failures across the run; this does not appear to be at all flaky.
2018-10-03 21:03:25 +01:00
Mikkel Nygaard Ravn a600fe7f13 Support materializing Flutter module host app on iOS (#21276)
* Prototype

* Fix paths to Flutter library resources

* Invoke pod install as necessary for materialized modules

* Add devicelab test for module use on iOS

* Remove debug output

* Rebase, reame materialize editable

* Add devicelab test editable iOS host app

* Removed add2app test section
2018-09-25 15:21:13 -04:00
Ian Hickson 9f359aeee2
Remove all "flaky" annotations from devicelab (#21895)
These tests don't seem particularly more flaky than ony others any more.
2018-09-17 13:02:34 -07:00
Jonah Williams c6e8a513f0
Add android semantics integration test to device lab (#20971) 2018-09-06 12:31:18 -07:00
amirh 146b906bbb
Mark the embedded_android_views_test as non flaky. (#20962)
Was marked as flaky as that's our policy for assing new device lab
tests. It's been consistently green so marking as non flaky.
2018-08-23 19:01:28 -07:00
amirh 1f31c3b354
Run the embedded Android views integration test on the device lab. (#20841) 2018-08-23 10:26:00 -07:00
Jonah Williams f0e80933b0
Mark memory_nav as flaky (#20930) 2018-08-22 17:34:29 -07:00
Jonah Williams 452f255e95
undo changes to devicelab capabilities (#20536) 2018-08-13 13:26:32 -07:00
Jonah Williams 0892ace91e
agent capability doesnt work as expected (#20529) 2018-08-13 10:19:47 -07:00
Jonah Williams cf88359ae4
restrict analysis task to faster machine (#20446) 2018-08-11 08:22:38 -07:00
liyuqian e770685a4b
Mark complex_layout_scroll_perf__timeline_summary as non-flaky (#20272) 2018-08-08 10:25:36 -07:00
Greg Spencer e60087a1a7
Remove Travis configuration from flutter/flutter (#20288)
This removes the final traces of Travis and Appveyor from the Flutter tree.

I've updated the documentation and fixed a couple of places where scripts look for Travis, and eliminated the dart tools runningOnTravis function (which was unused anyhow).

There are places in the flutter script that used to look for the environment variable TRAVIS. We actually do want to continue to detect that we're running on Travis there, since in the plugins repo we still use Travis (for the moment). In any case, it's OK, because the CI environment variable is set on all of the CI bots (Cirrus, Travis, and Appveyor).

FastLane doesn't have a setup_cirrus equivalent to setup_travis, but it actually doesn't matter there either, since it doesn't do Travis-specific things, and it also looks for the CI environment variable.
2018-08-07 13:41:33 -07:00
liyuqian b6cd38d8df
Add tiles_scroll_perf__timeline_summary to devicelab (#20105) 2018-08-02 16:47:58 -07:00
Ryan Macnak 091da9c7d3 Revert "mark flaky benchmarks as flaky (#19492)"
This reverts commit 7a9dd709c6.

Bug: https://github.com/flutter/flutter/issues/19490
2018-08-01 14:15:22 -07:00
Chris Bracken c0294e3a02
Eliminate Dart 1 benchmarks (#19430)
Flutter app development has been Dart 2 only for a significant amount of
time, and we're about to remove all support for Dart 1.
2018-07-18 15:56:51 -07:00
Michael Goderbauer 7a9dd709c6 mark flaky benchmarks as flaky (#19492) 2018-07-18 11:11:39 -07:00
Mikkel Nygaard Ravn ca0e144f9f
Mark non-flaky tests as such (#18927) 2018-06-29 10:55:31 +02:00
Danny Tuppeny e87b83bfd8 Mark flutter_gallery_instrumentation_test as flaky (#18887)
See #18879.
2018-06-27 15:48:22 -07:00
Ian Hickson 3ecd4301bf
Test performance of "flutter test" (#18810) 2018-06-26 17:34:25 -07:00
Ian Hickson 73498a54bc
Make flaky test as such. (#18815) 2018-06-25 19:25:01 -07:00
Mikkel Nygaard Ravn d4db748047
Add2app devicelab test (#18795) 2018-06-25 22:37:47 +02:00
Chris Bracken 7213a7805a
Mark flutter_attach_test flaky (#18801)
Introduced in d248725e15, but has been red
since landing. Marking flaky rather than reverting since it's the only
failure.
2018-06-25 11:04:51 -07:00
Florian Loitsch d248725e15
Flutter attach (#18677)
Add `flutter attach` command.

This command allows Flutter to connect to applications that haven't been launched using `flutter run`.
2018-06-25 15:33:42 +02:00
Michael Goderbauer ebb174b0bf
Add size benchmark for hello_world (#18522) 2018-06-16 00:08:33 -07:00
Ian Hickson d1cc8b6de8
Clean up output of "flutter run --release" (#18380)
(second attempt)
2018-06-14 12:30:20 -07:00