Deno.runTests() interface is not yet good enough to be exposed
publicly with stability guarantees.
This commit removes public API related to testing: Deno.runTests()
and Deno.TestMessage, but keeps them exposed on Deno.internal object
so they can be used with "deno test" subcommand.
* Reduce "testing" interfaces
* Use a callback instead of a generator for Deno.runTests()
* Default RunTestsOptions::reportToConsole to true
* Compose TestMessage into a single interface
This PR brings assertOps and assertResources sanitizers to Deno.test() API.
assertOps checks that test doesn't leak async ops, ie. there are no unresolved
promises originating from Deno APIs. Enabled by default, can be disabled using
Deno.TestDefinition.disableOpSanitizer.
assertResources checks that test doesn't leak resources, ie. all resources used
in test are closed. For example; if a file is opened during a test case it must be
explicitly closed before test case finishes. It's most useful for asynchronous
generators. Enabled by default, can be disabled using
Deno.TestDefinition.disableResourceSanitizer.
We've used those sanitizers in internal runtime tests and it proved very useful in
surfacing incorrect tests which resulted in interference between the tests.
All tests have been sanitized.
Closes#4208
This commit changes output of default test reporter to resemble output from Rust test runner;
first the name of running test is printed with "...", then after test has run result is printed on the same line.
* Split "Deno.TestEvent.Result" into "TestStart" and "TestEnd";
* changes TestReporter interface to support both events;
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
- moves compiler implementation to "cli/js/compiler/" directory
- moves more APIs to "cli/js/web":
* "console.ts"
* "console_table.ts"
* "performance.ts"
* "timers.ts"
* "workers.ts"
- removes some dead code from "cli/js/"
Following JS ops were moved to separate files in cli/js/ops directory:
- compiler
- dispatch_json
- dispatch_minimal
- errors
- fetch
- fs_events
- os
- random
- repl
- resources
- runtime_compiler
- runtime
- tty
* refactor: preliminary cleanup of Deno.runTests()
* Change time measurement to use new Date() instead of
performance.now(). Because there is no guarantee that tests are
run with "--allow-hr" using new Date() guarantees higher
precision of 1ms instead of 2ms.
* Support String type filter in "skip" and "only".
* Split "exitOnFail" into "exitOnFail" and "failFast".
Former tells if "runTests()" should exit with code 1 on test
failure, while latter tells if "runTests()" should stop
running tests on first failure.
* Use "defer" to wait for unhandled promise rejection - this bit
is funky and doesn't seem right, but for now it's just a rewrite
from using "setTimeout". Intended to be fixed in later commits.
* Remove global "__DENO_TEST_REGISTRY", don't expose list of
registered tests (to be addressed in follow up commits)
* Remove arbitrary slow test threshold; use uniform coloring
instead
* rewrite test runner in Rust
* migrate "test" and "runTests" functions from std to "Deno" namespace
* use "Deno.test()" to run internal JS unit tests
* remove std downloads for Deno subcommands
