Automatic set coverage when loading snapshot
TEST=Existing tests, manual inspection of reduction in sdk snapshot sizes (see data in comments in https://dart-review.googlesource.com/c/sdk/+/370501)
Change-Id: I044616144c2defeed252a6715eba1abcabffc86d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/371700
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The agnostic mode was added to allow the platform dill embedded in
the VM to support both weak and strong mode. Since weak mode is no
longer supported in the VM, the agnostic mode can new be deleted.
All uses of the agnostic in Dart and Flutter have been removed prior
to this change.
Change-Id: Iff0f69d9cd64e887e01cd7e7d336a97761bd6d4b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366801
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
This is a reland of commit b3253d9c8b
Fix the gn rules so the RBE offloading only happens in the Dart build
and the conflicting RBE configuration is not included otherwise. This
ensures this change can roll into Flutter while we design a better
solution to compose the projects' RBE usages.
Original change's description:
> [infra] RBE all the things.
>
> This change offloads all Dart actions during the build to RBE using
> the new rewrapper_dart.py script that understands all the different
> command line invocations of Dart programs and translates it into the
> appropriate rewrapper invocation, with a full list of input and output
> files and no absolute paths.
>
> The dart actions are all considered to be expensive RBE compilation
> steps that may be much slower on RBE than locally due to the low
> parallelism of the build during these stages and the comperatively
> slower bots in the RBE pools. The build uses the racing strategy by
> default for these compilations, such that cache hits are still used
> if available, and otherwise a local build is transparently used if it
> happens to be faster. The bots will use the remote strategy by default
> unlike developers, such that the rewrapper_dart.py script does not fail.
>
> rewrapper_dart.py contains a tiny dart import proprocessor written
> in python and a big argument parser that understands every command
> invoked during the build and the semantic meaning of every option.
>
> The absolute paths used during the Dart SDK build is not solved in
> this changelist, which just works around them initially, but these
> will be fixed in follow up changes with the appropriate teams now
> that this change proves they are not needed.
>
> Bug: b/333595242
> Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
> Reviewed-by: William Hesse <whesse@google.com>
> Commit-Queue: Jonas Termansen <sortie@google.com>
Bug: b/333595242
Change-Id: I43bec2e7fcad375e972ca447aa13ae65b0ce09ae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362661
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
This reverts commit b3253d9c8b.
Reason for revert: Flutter build is broken, it also has a GN variable rbi_dir, and dart_action.gni now imports two conflicting definitions.
Original change's description:
> [infra] RBE all the things.
>
> This change offloads all Dart actions during the build to RBE using
> the new rewrapper_dart.py script that understands all the different
> command line invocations of Dart programs and translates it into the
> appropriate rewrapper invocation, with a full list of input and output
> files and no absolute paths.
>
> The dart actions are all considered to be expensive RBE compilation
> steps that may be much slower on RBE than locally due to the low
> parallelism of the build during these stages and the comperatively
> slower bots in the RBE pools. The build uses the racing strategy by
> default for these compilations, such that cache hits are still used
> if available, and otherwise a local build is transparently used if it
> happens to be faster. The bots will use the remote strategy by default
> unlike developers, such that the rewrapper_dart.py script does not fail.
>
> rewrapper_dart.py contains a tiny dart import proprocessor written
> in python and a big argument parser that understands every command
> invoked during the build and the semantic meaning of every option.
>
> The absolute paths used during the Dart SDK build is not solved in
> this changelist, which just works around them initially, but these
> will be fixed in follow up changes with the appropriate teams now
> that this change proves they are not needed.
>
> Bug: b/333595242
> Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
> Reviewed-by: William Hesse <whesse@google.com>
> Commit-Queue: Jonas Termansen <sortie@google.com>
Bug: b/333595242
Change-Id: I56c9df11e672e7bd096f2a6e7d0243dccd0396c9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362385
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This change offloads all Dart actions during the build to RBE using
the new rewrapper_dart.py script that understands all the different
command line invocations of Dart programs and translates it into the
appropriate rewrapper invocation, with a full list of input and output
files and no absolute paths.
The dart actions are all considered to be expensive RBE compilation
steps that may be much slower on RBE than locally due to the low
parallelism of the build during these stages and the comperatively
slower bots in the RBE pools. The build uses the racing strategy by
default for these compilations, such that cache hits are still used
if available, and otherwise a local build is transparently used if it
happens to be faster. The bots will use the remote strategy by default
unlike developers, such that the rewrapper_dart.py script does not fail.
rewrapper_dart.py contains a tiny dart import proprocessor written
in python and a big argument parser that understands every command
invoked during the build and the semantic meaning of every option.
The absolute paths used during the Dart SDK build is not solved in
this changelist, which just works around them initially, but these
will be fixed in follow up changes with the appropriate teams now
that this change proves they are not needed.
Bug: b/333595242
Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
Automatically detect the appropriate configuration file if RBE=1 is
set to request RBE on all projects, or if DART_RBE=1 is set to request
RBE for Dart only, or otherwise respect the explicit configuration file.
Automatically set the server_address location to the build directory
if it has not already been set. This is unfortunately not supported on
Windows due to the environment variable being set during build but it
needed to be set during gn.
Retain the older configuration files during the transitory period.
Bug: b/296994239
Fixes: b/320876546
Change-Id: I62d1fbfed35248477731cceda3f7267c605c4969
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/355400
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Place the windows toolchain in the build root so it can be sent to RBE.
Use relative paths in debug symbols when building with Windows clang,
which is needed for remotely building with RBE.
Set the executable bit when invoking clang-cl remotely as it has not
been set on Windows RBE clients.
Stop including .rst files during build commands as @options-from-file
is not supported by rewrapper at this time.
Forward RBE_cfg into the msvc build environment and rewrite the
INCLUDE, LIB, and LIBPATH environment variables to use relative paths
instead of absolute paths.
Don't download reclient on windows-arm64 where there is no cipd
package and the windows-arm64 python lies and says it's x64 because
they actually packaged a x64 python.
Tune the Windows -j performance to match the historic bot performance
as most casual Windows machines won't handle -j 1000 well.
Bug: b/296994239
Change-Id: I2bd1b4478fca796f0a326c062a5f478f16c2c6ef
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/343440
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
Bug: b/296994239
Change-Id: Ie8630bcece5c2f62c5693afe654fb9d6367b33e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345286
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This change migrates the Dart linux builders from Goma to RBE via
reclient. Goma is being shut down by the end of the year and we must
migrate to the new replacement, which is much more powerful and can do
things we could never do with Goma. We should already see some mild
build time improvements from simply switching.
This change can be safely reverted if there are any problems:
- If you must revert this change, please reach out with what you
observed, but please first read and try the below instructions.
- Cross-compilation with RBE is not implemented at this time and certain
special builders running on Linux intel may not support RBE. The
existing configuration support to disable Goma remains in place. The
existence of configuration file makes the recipe switch all Linux x64
bots use RBE rather than Goma. Try disabling Goma on any builders that
are failing, whether using `--no-goma` in the test matrix or using
`goma = False` on infra/config. Goma is going away soon and we will
soon bring RBE up to feature parity and Goma isn't really working on
the special cross builds anyway.
- Rare flakes in the remote builds have previously been observed, but I
haven't seen them reappear yet after upgrading reclient. Look for
build log messages mentioning the remote execution failed, or that an
upload blob size digest mysteriously changed. Please alert me if you
see this and I will talk to the Foundry team. If this happens, please
try if you can fix this using `exec_strategy=remote_local_fallback` in
the configuration file. Right now reclient only does remote executions
to catch cases where RBE doesn't work, but if it fails, it can be
configured to silently fall back on a slow local build.
- If RBE is used on a configuration that doesn't support RBE builds, the
default -j 1000 may overload the bot if it falls back on a local
build. If this happens, the bot may explode and fail with BOT_DIED. If
this happens, disable Goma (RBE) on the builder as above and let me
know about the problem.
- We haven't tried sending a full production load to the RBE pool yet.
Flutter is already using the RBE pool partially. It's unclear what
will happen once we turn this on for everyone. Keep an eye out for
anything unusual and let me know.
- If it turns out RBE doesn't cache invalidate properly, it is possible
outdated object files will be linked. If anything really weird
happens, this may the case. As far as I tested, the caching is working
correctly.
RBE is yet to be officially rolled out to our developers. Stay tuned.
Bug: b/296994239
Change-Id: I77e0025197ab98a672efbc0cb04c8ece528c3e29
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335826
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
