After this removal, the only thing in tools/testing is the checked-in
jsshell (Mozilla command-line JavaScript runner) executable. Discussions
for removing that as well have been started.
Change-Id: I28505154de9fa7df3a2d5f84f3f778c596f52fcc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/106087
Reviewed-by: Bob Nystrom <rnystrom@google.com>
This makes it an actual Pub package like most other code inside the SDK
repo. The main goal is to make it easier to write tests for the test
runner itself.
This change:
- Moves all of the code from tools/testing/dart/ over to
pkg/test_runner. Most of it ends up under test_runner/lib/src.
- Move tools/testing/dart/main.dart to
pkg/test_runner/bin/test_runner.dart.
- Move standalone_2/io/test_runner_test.dart to
pkg/test_runner/test/test_runner_test.dart. I don't think it currently
works, but it wasn't being run in its old location either.
- Add test_runner to the analysis-server bot. This ensures the
test_runner package is static error clean.
- Remove standalone_2/io/test_runner_analyze_test.dart which used to
attempt to do the above and is no longer needed.
- Update test.py to look for the test runner at its new location.
- Add test_runner to the repo .packages file and remove the weird
test_dart pseudo-package. (I think this fixes #35279.)
- Remove status file entries for the removed standalone_2 tests.
There are no code changes to the test runner itself aside from fixing
up import paths.
Change-Id: I3d05d50d222b291848fa5a30de2846e803bc81e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105821
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
This was a leftover from a refactoring that was accidentally committed.
Change-Id: Ibbf26cabbfa16d1aba7fe4811977008c46c19c34
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104002
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This adds support for BabelJS to test.py's dart2js compiler
configuration. When --babel is specified directly or through a named
configuration, test.py will run an additional step after dart2js that
post-processes the javascript output by running it through BabelJS with
the specified Babel configuration. BabelJS is added to the DEPS in its
standalone form. d8 is used to run BabelJS standalone to avoid adding
a dependency on NodeJS. d8 can only write to stdout but not to files or
stderr, which makes it necessary to change the test_runner to handle
commands that expect their output to be piped to a file.
Changes:
* Add --babel option to test.py.
* Add babel option to pkg/smith.
* Switch IE11 builder to use babel transformation.
* Fix option list comparison bugs in pkg/smith.
* Change dart2js compiler configuration to generate files using the
test name rather than just "out.js" (update test that relied on this).
* Remove runtime_configuration dependency on test_suite.
* Remove obsolete blocks adding --preview-dart-2 dart2js arguments.
* Make dart2js' compiler configuration more like DDC's.
* Remove createCommand method that is no longer used.
* Remove support for "OtherResources" which was only used for
dart:isolate tests on dart2js and DDC.
* Skip co19_2 tests that are slow to transform with babel.
* Simplify the timeout handling in the test runner with Future.timeout.
Change-Id: I32e4917b2a57ecbe684538e40d744f0101c552a0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/90402
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
This is variant of SIMARM build (ARM target and non-ARM host) but with a
word size mismatch (32-bit target and 64-bit host). We only expect to
build gen_snapshot binary in this mode.
This is the first step towards enabling AOT compilation targeting ARM
using 64-bit gen_snapshot binary.
This change also introduces --gen-snapshot flag for test.py which allows
to specify which gen_snapshot binary should be used for running tests.
Expected workflow with SIMARM_X64 build:
$ tools/build.py -a simarm_x64 -m release gen_snapshot
$ tools/build.py -a simarm -m release dart_precompiled_runtime
vm_platform
$ tools/test.py -a simarm -m release -c dartkp --gen-snapshot
out/ReleaseSIMARM_X64/gen_snapshot
Note that our ARM simulator can't be built as a 64-binary so we are
going to be using SIMARM runtime to test AOT compiled code produced by
SIMARM_X64 binary.
Issue https://github.com/dart-lang/sdk/issues/36839
Issue https://github.com/flutter/flutter/issues/22598
Change-Id: Id003900e8b46fc9d57975ab82b0c21852a176079
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/100968
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Due to not having support for __VA_OPT__ yet the CL introduces a new
wet of macros ..._WITH_EXPECTATIONS() which can be given an expectation
marker.
Change-Id: I33812937f1b226fa89b3ab17a8a3483914abf2e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/100643
Reviewed-by: Stevie Strickland <sstrickl@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This reverts commit 7cb73d97d2.
Reason for revert: It caused windows build to fail (reason unknown):
c:\b\s\w\ir\cache\builder\sdk\runtime\vm\compiler\assembler\assembler_x64_test.cc(1097): error C3861: 'AssemblerTestGenerate_BitScanReverse': identifier not found
c:\b\s\w\ir\cache\builder\sdk\runtime\vm\compiler\assembler\assembler_x64_test.cc(5499): error C3861: 'AssemblerTestGenerate_bittest': identifier not found
Original change's description:
> [Infra] Let run_vm_tests --list output a test expectation marker
>
> Due to not having support for __VA_OPT__ yet the CL introduces a new
> wet of macros ..._WITH_EXPECTATIONS() which can be given an expectation
> marker.
>
> Change-Id: Icaac4672f04340fe4644d13a14c32704ba36daec
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96940
> Commit-Queue: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Jonas Termansen <sortie@google.com>
> Reviewed-by: Régis Crelier <regis@google.com>
TBR=sortie@google.com,kustermann@google.com,regis@google.com
Change-Id: Ica07ef5e7302e3d8e4b131427ff5ab20be8c6a63
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/100581
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Due to not having support for __VA_OPT__ yet the CL introduces a new
wet of macros ..._WITH_EXPECTATIONS() which can be given an expectation
marker.
Change-Id: Icaac4672f04340fe4644d13a14c32704ba36daec
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96940
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
This switches the "dartdevk" test configuration to use Kernel IL files
directly, rather than automatically converting Analyzer summaries.
Change-Id: Ifeeff98f298050b206c17678a85ee87909d41f46
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/99370
Commit-Queue: Jenny Messerly <jmesserly@google.com>
Auto-Submit: Jenny Messerly <jmesserly@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Closing the DebugLogger ensures that any buffered messages are flushed
before mulittest.dart calls exit().
Change-Id: If55629bae2fcc518c566c60ba4bd9aa9fddae65e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98678
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
The two existing places where IR is built are changed to use this new
helper. It supports running normal JIT/AOT passes as well as a
user-specified set of passes.
In order to allow vm/cc tests to make assertions about AOT pipeline this CL
enables the DART_PRECOMPILER define in run_vm_tests binary (similar to
gen_snapshot, run_vm_tests has now JIT and AOT support)
Change-Id: Ib51a024a81e0291e89d20860b8b9a2762611426c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98482
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Aart Bik <ajcbik@google.com>
The fix for https://github.com/dart-lang/sdk/issues/35960 broke
on app_jit(k) runs, because these expect the arguments to be
passed when creating the jit snapshot. Pass dartOptions along to
computeCompilerArguments as well, so that the
AppJitCompilerConfiguration can add them in appropriately.
Change-Id: I8c7b5a3a1689943db4e6c3785ccaf8bfdb839c2d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97516
Reviewed-by: Stevie Strickland <sstrickl@google.com>
Commit-Queue: Stevie Strickland <sstrickl@google.com>
* Move FFI tests into a separate test suite.
They never belonged in standalone_2/ since they are not only available in
the standalone VM. Also, we want to have a separate status file.
* Add new "SharedObjects" option to test files to copy needed shared objects
to the Android device for testing.
* Add support to compiler/runtime_configuration.dart for testing JIT-mode on Android.
* Add new configurations and builders to test_matrix.json to test JIT-mode on Android.
* Clean up status file entries for FFI (we didn't need to special-case stress & subtype tests).
Change-Id: Ifb32ef7051754f477d00ecd7a0f9b19ca8a66eae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97334
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Only pass DartOptions to the runtime, not to the compiler.
Since commit 794e1e3a09, `gen_snapshot` now attempts to
use these extra arguments (and fails), whereas before it
ignored them.
This fixes https://github.com/dart-lang/sdk/issues/35960.
Change-Id: I30cfcd1f5e10fe0348eef7d42d9e721a61e818cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97507
Reviewed-by: William Hesse <whesse@google.com>
Auto-Submit: Stevie Strickland <sstrickl@google.com>
Commit-Queue: Stevie Strickland <sstrickl@google.com>
1. Remove support for the following flags '-c', '--checked' and '--enable-checked-mode'
2. Cleanup some of the tests and test scripts where these options were being passed.
https://github.com/dart-lang/sdk/issues/34660
Change-Id: I4d8aa0d14bd054cfba08d78a411a0df4fc829df1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97550
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
All configurations are created with the same named configuration,
if there is a named configuration argument.
Change-Id: I2de34a4a503c34e6b24d3edcf0dd06567ed4f0ed
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97327
Reviewed-by: Jonas Termansen <sortie@google.com>
This option makes the available named configurations much more discoverable.
Additionally this change expands the available named configurations by
generalizing the operating system and processor architecture patterns
in the named configurations. This change should ensure that nobody is doing
any local testing that isn't covered by a named configuration.
Change-Id: I776105955a86e9f0403ce07a3cdf971e4213646f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96320
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Previously, messages would contain "Warning:" twice and failed to print
which file the errors came from.
Removed newlines from messages because the logger only prints the first
line.
Fixes#34405
Change-Id: I34d3679a4d769d7c7e2488ca55ffd15f08af9b16
Reviewed-on: https://dart-review.googlesource.com/c/93990
Commit-Queue: Alexander Thomas <athom@google.com>
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
All of DDC's supported platforms have Promises, so we can use them
instead of MutationObservers (web) and timers (node.js).
See issue #20055 (same issue, but for dart2js).
Change-Id: Id635a4a9fa104a2ab19dd20824d209f682f831f9
Reviewed-on: https://dart-review.googlesource.com/c/91765
Reviewed-by: Mark Zhou <markzipan@google.com>
Commit-Queue: Jenny Messerly <jmesserly@google.com>
Specify extra shared options to the compiler and runtime via the
--shared-options= commandline option to test.py. For instance,
--shared-options=--enable-experiment=constant-update-2018 will enable
the front-end constant evaluation experiment for all tests run.
To avoid conflicts with tests explicitly enabling experiments, the
front-end parsing of experimental flags is changed to allow the same
flag to be mentioned several times, as long as the value is the same.
This matches the existing behavior of the analyzer.
Change-Id: I0c79e62b0b23ec7bc3824064dd27ef86b4451f9a
Reviewed-on: https://dart-review.googlesource.com/c/93501
Commit-Queue: Aske Simon Christensen <askesc@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Change-Id: Ibfb2c4542750b7cc52672c3e180adadd52bf0848
Reviewed-on: https://dart-review.googlesource.com/c/93500
Commit-Queue: Alexander Thomas <athom@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Also fixes 2 html tests whose failure was caught by this change.
Change-Id: I707cba922bc7e7728ba0b026ad056b547fcd1432
Reviewed-on: https://dart-review.googlesource.com/c/92392
Auto-Submit: Jenny Messerly <jmesserly@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
This fixes a bug in test.py's new --tests option.
Change-Id: I0410972116045c612429565081ad84735d2d2bf1
Reviewed-on: https://dart-review.googlesource.com/c/92784
Auto-Submit: Alexander Thomas <athom@google.com>
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Previously, this only file names were allowed. However, deflaking may
run on a different machine than determinining which tests flaked. Allow
callers to specify a list of tests instead of a file, simplifies sharded
deflaking considerably.
Change-Id: I710f9aa8a8ef696a6982851106c0e171343dc190
Reviewed-on: https://dart-review.googlesource.com/c/92561
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
This change ensures that DDC tests won't accidentally "pass", and then
silently fail or timeout later, when the async tests actually run.
Change-Id: Id18d3343e5dadcf266f39568a93381daf4c54e1c
Reviewed-on: https://dart-review.googlesource.com/c/92247
Reviewed-by: Mark Zhou <markzipan@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Commit-Queue: Jenny Messerly <jmesserly@google.com>
Plumb experimental flags found in a test's SharedOptions through to the
gen_kernel command invoked in dartkp tests. Also enable a whole bunch
of tests that are now passing.
Bug: https://github.com/dart-lang/sdk/issues/35602
Change-Id: I425f55ea6c84d28e085ab2c848376e396e68a225
Reviewed-on: https://dart-review.googlesource.com/c/90860
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Support for the flutter runtime (sky_shell) was added to the Dart
test framework in the hope that one could run the flutter engine
through the Dart test suite. We never got around to running these
tests on the build bots, the sky_shell executable has been deprecated
and there are plans in the flutter engine team to have their own
unit test frame work.
See Issue https://github.com/flutter/flutter/issues/9115 for more
details.
Change-Id: I6ee9e8c919721dccecf4202ab0778939aeb79174
Reviewed-on: https://dart-review.googlesource.com/c/91103
Reviewed-by: Ben Konyi <bkonyi@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
All multitests expecting errors were just being reported as expecting "fail", not as expecting compile-time
or runtime error. Cleaned up legacy checked-only expectations as a side effect.
Change-Id: I7785db0e094e3af3464700fbc71aac30b8bb4282
Reviewed-on: https://dart-review.googlesource.com/c/90684
Reviewed-by: Jonas Termansen <sortie@google.com>
Auto-Submit: William Hesse <whesse@google.com>
This reverts commit 8044e32fc5.
Reason for revert: The device was fixed, and there is now one device per bot. Blacklisting a single device is no longer the right approach because it effectively destroys a bot.
Original change's description:
> [infra] Blacklist android device id in attempt to make buildbot builder more stable
>
> This particular device sometimes fails with errors such as:
>
> * "protocol error"
> * "protocol error (no status)"
> * "device not found"
>
> Change-Id: Iced97ae8f5e891c43436a3d745004517dc59b42c
> Reviewed-on: https://dart-review.googlesource.com/c/84905
> Reviewed-by: Alexander Thomas <athom@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
> Commit-Queue: Martin Kustermann <kustermann@google.com>
TBR=whesse@google.com,vegorov@google.com,kustermann@google.com,athom@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I8422a75d10518ec06de737a0a719f9648279fbf8
Reviewed-on: https://dart-review.googlesource.com/c/89500
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Keeps comment that many test options should not be used in co19 tests.
Change-Id: Iacdc6a8d240d8fd5d6ac87fc5a16ff38978d2040
Reviewed-on: https://dart-review.googlesource.com/c/88704
Reviewed-by: Alexander Thomas <athom@google.com>
