mirror of
https://github.com/flutter/flutter
synced 2024-08-25 02:46:31 +00:00
Document how to read the dashboard (#14577)
* Document how to read the dashboard * address comments
This commit is contained in:
parent
6d2dc050b2
commit
e3eaf04c6c
|
@ -7,15 +7,91 @@ This package contains the code for test framework and the tests. More generally
|
|||
the tests are referred to as "tasks" in the API, but since we primarily use it
|
||||
for testing, this document refers to them as "tests".
|
||||
|
||||
If you have access to Google's internal network, you can see the continuous
|
||||
build results from the master branch at <http://go/flutter-dashboard/build.html>.
|
||||
(There is currently no public view of this data, unfortunately.)
|
||||
Build results are available at https://flutter-dashboard.appspot.com.
|
||||
|
||||
# Prerequisites
|
||||
# Reading the dashboard
|
||||
|
||||
You must set the `ANDROID_HOME` environment variable to run tests on Android. If
|
||||
you have a local build of the Flutter engine, then you have a copy of the
|
||||
Android SDK at `.../engine/src/third_party/android_tools/sdk`.
|
||||
## The build page
|
||||
|
||||
The build page is accessible at https://flutter-dashboard.appspot.com/build.html.
|
||||
This page reports the health of build servers, called _agents_, and the statuses
|
||||
of build tasks.
|
||||
|
||||
### Agents
|
||||
|
||||
A green agent is considered healthy and ready to receive new tasks to build. A
|
||||
red agent is broken and does not receive new tasks.
|
||||
|
||||
In the example below, the dashboard shows that the `linux2` agent is broken and
|
||||
requires attention. All other agents are healthy.
|
||||
|
||||
![Agent statuses](images/agent-statuses.png)
|
||||
|
||||
### Tasks
|
||||
|
||||
The table below the agent statuses displays the statuses of build tasks. Task
|
||||
statuses are color-coded. The following statuses are available:
|
||||
|
||||
**New task** (light blue): the task is waiting for an agent to pick it up and
|
||||
start the build.
|
||||
|
||||
**Task is running** (spinning blue): an agent is currently building the task.
|
||||
|
||||
**Task succeeded** (green): an agent reported a successful completion of the
|
||||
task.
|
||||
|
||||
**Task is flaky** (yellow): the task was attempted multiple time, but only the
|
||||
latest attempt succeeded (we currently only try twice).
|
||||
|
||||
**Task failed** (red): the task failed all of the attempts.
|
||||
|
||||
**Task underperformed** (orange): currently not used.
|
||||
|
||||
**Task was skipped** (transparent): the task is not scheduled for a build. This
|
||||
usually happens when a task is removed from `manifest.yaml` file.
|
||||
|
||||
**Task status unknown** (purple): currently not used.
|
||||
|
||||
In addition to color-coding, a task may display a question mark. This means
|
||||
that the task was marked as flaky manually. The status of such task is ignored
|
||||
when considering whether the build is broken or not. For example, if a flaky
|
||||
task fails, GitHub will not prevent PR submissions. However, if the latest
|
||||
status of a non-flaky task is red, all pending PRs will contain a warning about
|
||||
the broken build and recommend caution when submitting.
|
||||
|
||||
Legend:
|
||||
|
||||
![Task status legend](images/legend.png)
|
||||
|
||||
The example below shows that commit `e122d5d` caused a wide-spread breakage,
|
||||
which was fixed by `bdc6f10`. It also shows that Travis, AppVeyor and Chrome
|
||||
Infra (left-most tasks) decided to skip building these commits. Hovering over
|
||||
a cell will pop up a tooltip containing the name of the broken task. Clicking
|
||||
on the cell will open the log file in a new browser tab (only visible to core
|
||||
contributors as of today).
|
||||
|
||||
![Broken Test](images/broken-test.png)
|
||||
|
||||
## Why is a task stuck on "new task" status?
|
||||
|
||||
The dashboard aggregates build results from multiple build environments,
|
||||
including Travis, AppVeyor, Chrome Infra, and devicelab. While devicelab
|
||||
tests every commit that goes into the `master` branch, other environments
|
||||
may skip some commits. For example, Travis and AppVeyor will only test the
|
||||
_last_ commit of a PR that's merged into the `master` branch. Chrome Infra may
|
||||
skip commits when they come in too fast.
|
||||
|
||||
## How the devicelab runs the tasks
|
||||
|
||||
The devicelab agents have a small script installed on them that continuously
|
||||
asks the CI server for tasks to run. When the server finds a suitable task for
|
||||
an agent it reserves that task for the agent. If the task succeeds, the agent
|
||||
reports the success to the server and the dashboard shows that task in green.
|
||||
If the task fails, the agent reports the failure to the server, the server
|
||||
increments the counter counting the number of attempts it took to run the task
|
||||
and puts the task back in the pool of available tasks. If a task does not
|
||||
succeed after a certain number of attempts (as of this writing the limit is 2),
|
||||
the task is marked as failed and is displayed using red color on the dashboard.
|
||||
|
||||
# Running tests locally
|
||||
|
||||
|
@ -24,6 +100,12 @@ Below is a handful of commands that run tests in a similar way to how the
|
|||
CI environment runs them. These commands are also useful when you need to
|
||||
reproduce a CI test failure locally.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You must set the `ANDROID_HOME` environment variable to run tests on Android. If
|
||||
you have a local build of the Flutter engine, then you have a copy of the
|
||||
Android SDK at `.../engine/src/third_party/android_tools/sdk`.
|
||||
|
||||
To run a test, use option `-t` (`--task`):
|
||||
|
||||
```sh
|
||||
|
@ -61,15 +143,6 @@ Currently there are only three stages defined, `devicelab`, `devicelab_ios` and
|
|||
|
||||
# Reproducing broken builds locally
|
||||
|
||||
If a commit caused a test to fail,
|
||||
[the dashboard](http://go/flutter-dashboard/build.html) (requires access to the
|
||||
Google network, sorry) might look something like this:
|
||||
|
||||
![Broken Test](images/broken-test.png)
|
||||
|
||||
The red circle tells you that a test failed. The number inside tells you how
|
||||
many times the devicelab attempted to run the test before giving up on it.
|
||||
|
||||
To reproduce the breakage locally `git checkout` the corresponding Flutter
|
||||
revision. Note the name of the test that failed. In the example above the
|
||||
failing test is `flutter_gallery__transition_perf`. This name can be passed to
|
||||
|
|
BIN
dev/devicelab/images/agent-statuses.png
Normal file
BIN
dev/devicelab/images/agent-statuses.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 68 KiB |
BIN
dev/devicelab/images/legend.png
Normal file
BIN
dev/devicelab/images/legend.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 48 KiB |
Loading…
Reference in a new issue