Document how to read the dashboard (#14577)

* Document how to read the dashboard

* address comments

dev/devicelab/README.md

@@ -7,15 +7,91 @@ This package contains the code for test framework and the tests. More generally
 the tests are referred to as "tasks" in the API, but since we primarily use it
 for testing, this document refers to them as "tests".
 
-If you have access to Google's internal network, you can see the continuous
-build results from the master branch at <http://go/flutter-dashboard/build.html>.
-(There is currently no public view of this data, unfortunately.)
+Build results are available at https://flutter-dashboard.appspot.com.
 
-# Prerequisites
+# Reading the dashboard
 
-You must set the `ANDROID_HOME` environment variable to run tests on Android. If
-you have a local build of the Flutter engine, then you have a copy of the
-Android SDK at `.../engine/src/third_party/android_tools/sdk`.
+## The build page
+
+The build page is accessible at https://flutter-dashboard.appspot.com/build.html.
+This page reports the health of build servers, called _agents_, and the statuses
+of build tasks.
+
+### Agents
+
+A green agent is considered healthy and ready to receive new tasks to build. A
+red agent is broken and does not receive new tasks.
+
+In the example below, the dashboard shows that the `linux2` agent is broken and
+requires attention. All other agents are healthy.
+
+![Agent statuses](images/agent-statuses.png)
+
+### Tasks
+
+The table below the agent statuses displays the statuses of build tasks. Task
+statuses are color-coded. The following statuses are available:
+
+**New task** (light blue): the task is waiting for an agent to pick it up and
+start the build.
+
+**Task is running** (spinning blue): an agent is currently building the task.
+
+**Task succeeded** (green): an agent reported a successful completion of the
+task.
+
+**Task is flaky** (yellow): the task was attempted multiple time, but only the
+latest attempt succeeded (we currently only try twice).
+
+**Task failed** (red): the task failed all of the attempts.
+
+**Task underperformed** (orange): currently not used.
+
+**Task was skipped** (transparent): the task is not scheduled for a build. This
+usually happens when a task is removed from `manifest.yaml` file.
+
+**Task status unknown** (purple): currently not used.
+
+In addition to color-coding, a task may display a question mark. This means
+that the task was marked as flaky manually. The status of such task is ignored
+when considering whether the build is broken or not. For example, if a flaky
+task fails, GitHub will not prevent PR submissions. However, if the latest
+status of a non-flaky task is red, all pending PRs will contain a warning about
+the broken build and recommend caution when submitting.
+
+Legend:
+
+![Task status legend](images/legend.png)
+
+The example below shows that commit `e122d5d` caused a wide-spread breakage,
+which was fixed by `bdc6f10`. It also shows that Travis, AppVeyor and Chrome
+Infra (left-most tasks) decided to skip building these commits. Hovering over
+a cell will pop up a tooltip containing the name of the broken task. Clicking
+on the cell will open the log file in a new browser tab (only visible to core
+contributors as of today).
+
+![Broken Test](images/broken-test.png)
+
+## Why is a task stuck on "new task" status?
+
+The dashboard aggregates build results from multiple build environments,
+including Travis, AppVeyor, Chrome Infra, and devicelab. While devicelab
+tests every commit that goes into the `master` branch, other environments
+may skip some commits. For example, Travis and AppVeyor will only test the
+_last_ commit of a PR that's merged into the `master` branch. Chrome Infra may
+skip commits when they come in too fast.
+
+## How the devicelab runs the tasks
+
+The devicelab agents have a small script installed on them that continuously
+asks the CI server for tasks to run. When the server finds a suitable task for
+an agent it reserves that task for the agent. If the task succeeds, the agent
+reports the success to the server and the dashboard shows that task in green.
+If the task fails, the agent reports the failure to the server, the server
+increments the counter counting the number of attempts it took to run the task
+and puts the task back in the pool of available tasks. If a task does not
+succeed after a certain number of attempts (as of this writing the limit is 2),
+the task is marked as failed and is displayed using red color on the dashboard.
 
 # Running tests locally
 
@@ -24,6 +100,12 @@ Below is a handful of commands that run tests in a similar way to how the
 CI environment runs them. These commands are also useful when you need to
 reproduce a CI test failure locally.
 
+## Prerequisites
+
+You must set the `ANDROID_HOME` environment variable to run tests on Android. If
+you have a local build of the Flutter engine, then you have a copy of the
+Android SDK at `.../engine/src/third_party/android_tools/sdk`.
+
 To run a test, use option `-t` (`--task`):
 
 ```sh
@@ -61,15 +143,6 @@ Currently there are only three stages defined, `devicelab`, `devicelab_ios` and
 
 # Reproducing broken builds locally
 
-If a commit caused a test to fail,
-[the dashboard](http://go/flutter-dashboard/build.html) (requires access to the
-Google network, sorry) might look something like this:
-
-![Broken Test](images/broken-test.png)
-
-The red circle tells you that a test failed. The number inside tells you how
-many times the devicelab attempted to run the test before giving up on it.
-
 To reproduce the breakage locally `git checkout` the corresponding Flutter
 revision. Note the name of the test that failed. In the example above the
 failing test is `flutter_gallery__transition_perf`. This name can be passed to