2016-02-11 20:43:05 +00:00
|
|
|
# Flutter's Build Infrastructure
|
2016-01-16 00:06:51 +00:00
|
|
|
|
2016-02-11 20:43:05 +00:00
|
|
|
This directory exists to support building Flutter on our build infrastructure.
|
2016-01-16 00:06:51 +00:00
|
|
|
|
2016-05-18 18:38:45 +00:00
|
|
|
The results of such builds are viewable at:
|
|
|
|
|
* https://build.chromium.org/p/client.flutter/waterfall
|
|
|
|
|
* https://travis-ci.org/flutter/flutter/builds (limited checking used just for PRs on github)
|
2016-01-16 00:06:51 +00:00
|
|
|
|
2016-05-18 18:38:45 +00:00
|
|
|
The external master pages for the chromium infra bots do not allow
|
|
|
|
|
forcing new builds. Contact @eseidelGoogle or another member of
|
|
|
|
|
Google's Flutter team if you need to do that.
|
2016-02-11 20:43:05 +00:00
|
|
|
|
2016-05-18 18:38:45 +00:00
|
|
|
The Travis-based bots are trivial, and just run a couple of shell
|
|
|
|
|
scripts. The rest of this document discusses only the chromium infra
|
|
|
|
|
bots.
|
|
|
|
|
|
|
|
|
|
This infrastructure is broken into two parts. A buildbot master specified by our
|
2016-02-11 20:43:05 +00:00
|
|
|
[builders.pyl](https://chromium.googlesource.com/chromium/tools/build.git/+/master/masters/master.client.flutter/builders.pyl)
|
|
|
|
|
file, and a [set of
|
|
|
|
|
recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter)
|
|
|
|
|
which we run on that master. Both of these technologies are highly specific to
|
|
|
|
|
Google's Chromium project. We're just borrowing some of their infrastructure.
|
|
|
|
|
|
2016-02-23 00:59:03 +00:00
|
|
|
## Prerequisites
|
|
|
|
|
|
|
|
|
|
- [install depot_tools](http://www.chromium.org/developers/how-tos/install-depot-tools)
|
|
|
|
|
- Python package installer: `sudo apt-get install python-pip`
|
|
|
|
|
- Python coverage package (only needed for `training_simulation`): `sudo pip install coverage`
|
|
|
|
|
|
|
|
|
|
## Getting the code
|
|
|
|
|
|
|
|
|
|
The following will get way more than just recipe code, but it _will_ get the recipe code:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
mkdir chrome_infra
|
|
|
|
|
cd chrome_infra
|
|
|
|
|
fetch infra
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
More detailed instructions can be found [here](https://chromium.googlesource.com/infra/infra/+/master/doc/source.md).
|
|
|
|
|
|
|
|
|
|
Most of the functionality for recipes comes from `recipe_modules`, which are
|
|
|
|
|
unfortunately spread to many separate repositories. After checking out the code
|
|
|
|
|
search for files named `api.py` or `example.py` under `infra/build`.
|
|
|
|
|
|
2016-02-11 20:43:05 +00:00
|
|
|
## Editing a recipe
|
|
|
|
|
|
|
|
|
|
Flutter has one recipe per repository. Currently
|
|
|
|
|
[flutter/flutter](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/flutter.py)
|
|
|
|
|
and
|
2016-04-18 19:31:18 +00:00
|
|
|
[flutter/engine](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/engine.py):
|
|
|
|
|
|
|
|
|
|
- build/scripts/slave/recipes/flutter/flutter.py
|
|
|
|
|
- build/scripts/slave/recipes/flutter/engine.py
|
2016-02-11 20:43:05 +00:00
|
|
|
|
2016-02-23 00:59:03 +00:00
|
|
|
Recipes are just Python. They are
|
2016-02-11 20:43:05 +00:00
|
|
|
[documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md)
|
|
|
|
|
by the [luci/recipes-py github project](https://github.com/luci/recipes-py).
|
|
|
|
|
|
2016-02-23 00:59:03 +00:00
|
|
|
The typical cycle for editing a recipe is:
|
2016-02-13 13:59:41 +00:00
|
|
|
|
2017-03-13 17:14:02 +00:00
|
|
|
1. Make your edits (probably to files in
|
|
|
|
|
`//chrome_infra/build/scripts/slave/recipes/flutter`).
|
2017-05-09 15:30:05 +00:00
|
|
|
2. Run `build/scripts/slave/recipes.py --use-bootstrap test train` to
|
|
|
|
|
update expected files
|
2017-03-13 17:14:02 +00:00
|
|
|
3. Run `build/scripts/tools/run_recipe.py flutter/<repo> slavename=<slavename>
|
2018-01-11 01:05:47 +00:00
|
|
|
mastername=client.flutter buildername=<buildername> buildnumber=1234` where `<repo>` is one
|
2017-03-13 17:14:02 +00:00
|
|
|
of `flutter` or `engine`, and `slavename` and `buildername` can be looked up
|
|
|
|
|
from the *Build Properties* section of a [recent
|
|
|
|
|
build](https://build.chromium.org/p/client.flutter/one_line_per_build).
|
|
|
|
|
4. Upload the patch (`git commit`, `git cl upload`) and send it to someone in
|
|
|
|
|
the `recipes/flutter/OWNERS` file for review.
|
2016-02-13 13:59:12 +00:00
|
|
|
|
2016-02-11 20:43:05 +00:00
|
|
|
## Editing the client.flutter buildbot master
|
|
|
|
|
|
|
|
|
|
Flutter uses Chromium's fancy
|
|
|
|
|
[builders.pyl](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
|
|
|
|
|
master generation system. Chromium hosts 100s (if not 1000s) of buildbot
|
|
|
|
|
masters and thus has lots of infrastructure for turning them up and down.
|
2016-02-23 00:59:03 +00:00
|
|
|
Eventually all of buildbot is planned to be replaced by other infrastructure,
|
|
|
|
|
but for now flutter has its own client.flutter master.
|
2016-02-11 20:43:05 +00:00
|
|
|
|
|
|
|
|
You would need to edit client.flutter's master in order to add slaves (talk to
|
|
|
|
|
@eseidelGoogle), add builder groups, or to change the html layout of
|
|
|
|
|
https://build.chromium.org/p/client.flutter. Carefully follow the [builders.pyl
|
|
|
|
|
docs](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
|
|
|
|
|
to do so.
|
|
|
|
|
|
|
|
|
|
## Future Directions
|
|
|
|
|
|
|
|
|
|
We would like to host our own recipes instead of storing them in
|
|
|
|
|
[build](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter).
|
|
|
|
|
Support for [cross-repository
|
|
|
|
|
recipes](https://github.com/luci/recipes-py/blob/master/doc/cross_repo.md) is
|
2016-02-23 00:59:03 +00:00
|
|
|
in-progress. If you view the git log of this directory, you'll see we initially
|
2016-02-11 20:43:05 +00:00
|
|
|
tried, but it's not quite ready.
|
2017-02-07 00:06:35 +00:00
|
|
|
|
|
|
|
|
# Android Tools
|
|
|
|
|
|
2018-01-05 20:40:17 +00:00
|
|
|
The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google Cloud. During the build a bot runs the
|
|
|
|
|
`download_android_tools.py` script that downloads the required version of the Android SDK into `dev/bots/android_tools`.
|
|
|
|
|
|
|
|
|
|
To check which components are currently installed, download the current SDK stored in Google Cloud using the
|
|
|
|
|
`download_android_tools.py` script, then `dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some
|
|
|
|
|
components need to be updated or installed, follow the steps below:
|
2017-02-07 00:06:35 +00:00
|
|
|
|
|
|
|
|
## How to update Android SDK on Google Cloud Storage
|
|
|
|
|
|
|
|
|
|
1. Run Android SDK Manager and update packages
|
|
|
|
|
`$ dev/bots/android_tools/sdk/tools/android update sdk`
|
|
|
|
|
Use `android.bat` on Windows.
|
|
|
|
|
|
2018-01-05 20:40:17 +00:00
|
|
|
2. Use the UI to choose the packages you want to install and/or update.
|
|
|
|
|
|
|
|
|
|
3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows, run `sdkmanager.bat` instead. If the
|
|
|
|
|
process fails with an error saying that it is unable to move files (Windows makes files and directories read-only
|
|
|
|
|
when another process is holding them open), make a copy of the `dev/bots/android_tools/sdk/tools` directory, run
|
|
|
|
|
the `sdkmanager.bat` from the copy, and use the `--sdk_root` option pointing at `dev/bots/android_tools/sdk`.
|
|
|
|
|
|
|
|
|
|
4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept the licenses for the newly installed
|
|
|
|
|
components. It also helps to run this command a second time and make sure that it prints "All SDK package licenses
|
|
|
|
|
accepted".
|
|
|
|
|
|
|
|
|
|
5. Run upload_android_tools.py -t sdk
|
2017-02-07 00:06:35 +00:00
|
|
|
`$ dev/bots/upload_android_tools.py -t sdk`
|
|
|
|
|
|
|
|
|
|
## How to update Android NDK on Google Cloud Storage
|
|
|
|
|
|
|
|
|
|
1. Download a new NDK binary (e.g. android-ndk-r10e-linux-x86_64.bin)
|
|
|
|
|
2. cd dev/bots/android_tools
|
|
|
|
|
`$ cd dev/bots/android_tools`
|
|
|
|
|
|
|
|
|
|
3. Remove the old ndk directory
|
|
|
|
|
`$ rm -rf ndk`
|
|
|
|
|
|
|
|
|
|
4. Run the new NDK binary file
|
|
|
|
|
`$ ./android-ndk-r10e-linux-x86_64.bin`
|
|
|
|
|
|
|
|
|
|
5. Rename the extracted directory to ndk
|
|
|
|
|
`$ mv android-ndk-r10e ndk`
|
|
|
|
|
|
|
|
|
|
6. Run upload_android_tools.py -t ndk
|
|
|
|
|
`$ cd ../..`
|
|
|
|
|
`$ dev/bots/upload_android_tools.py -t ndk`
|
