vscode/test/smoke/README.md

84 lines
3.9 KiB
Markdown
Raw Normal View History

2017-09-25 15:36:58 +00:00
# VS Code Smoke Test
2017-06-30 13:32:45 +00:00
2020-02-07 11:41:08 +00:00
Make sure you are on **Node v12.x**.
2018-05-03 09:52:10 +00:00
### Run
2018-05-03 09:49:28 +00:00
2018-10-23 18:36:58 +00:00
```bash
# Install Dependencies and Compile
yarn --cwd test/smoke
yarn --cwd test/automation
2019-04-01 09:10:13 +00:00
# Dev (Electron)
2018-04-18 10:01:17 +00:00
yarn smoketest
2017-09-04 09:32:41 +00:00
# Dev (Web)
yarn smoketest --web --browser [chromium|firefox|webkit]
2019-05-31 21:17:33 +00:00
# Build (Electron)
yarn smoketest --build <path latest built version> --stable-build <path to previous stable version>
# Build (Web - read instructions below)
yarn smoketest --build <path to web server folder> --web --browser [chromium|firefox|webkit]
# Remote (Electron)
yarn smoketest --build <path latest built version> --remote
2017-09-08 17:04:52 +00:00
```
2017-06-01 13:50:23 +00:00
2018-05-03 09:49:28 +00:00
### Run for a release
You must always run the smoketest version which matches the release you are testing. So, if you want to run the smoketest for a release build (e.g. `release/1.22`), you need that version of the smoke tests too:
2018-05-03 09:49:28 +00:00
2018-10-23 18:36:58 +00:00
```bash
2018-05-03 09:49:28 +00:00
git checkout release/1.22
yarn --cwd test/smoke
```
#### Electron
2019-05-22 11:01:41 +00:00
In addition to the new build to be released you will need the previous stable build so that the smoketest can test the data migration.
The recommended way to make these builds available for the smoketest is by downloading their archive version (\*.zip) and extracting
them into two folders. Pass the folder paths to the smoketest as follows:
```bash
yarn smoketest --build <path latest built version> --stable-build <path to previous stable version>
```
#### Web
**macOS**: if you have downloaded the server with web bits, make sure to run the following command before unzipping it to avoid security issues on startup:
```bash
xattr -d com.apple.quarantine <path to server with web folder zip>
2018-05-03 09:49:28 +00:00
```
2018-04-18 10:01:17 +00:00
There is no support for testing an old version to a new one yet, so simply configure the `--build` command line argument to point to
the web server folder which includes the web client bits (e.g. `vscode-server-darwin-web` for macOS).
**Note**: make sure to point to the server that includes the client bits!
2018-05-03 09:52:10 +00:00
### Debug
2017-06-01 13:50:23 +00:00
2018-05-03 09:45:39 +00:00
- `--verbose` logs all the low level driver calls made to Code;
- `-f PATTERN` (alias `-g PATTERN`) filters the tests to be run. You can also use pretty much any mocha argument;
2018-05-03 09:49:28 +00:00
- `--screenshots SCREENSHOT_DIR` captures screenshots when tests fail.
2018-05-02 09:33:49 +00:00
2018-05-03 09:52:10 +00:00
### Develop
2018-05-03 09:45:39 +00:00
2018-10-23 18:36:58 +00:00
```bash
2018-05-03 09:45:39 +00:00
cd test/smoke
yarn watch
```
2017-09-25 15:36:58 +00:00
## Pitfalls
2017-06-01 13:50:23 +00:00
2018-04-18 10:01:17 +00:00
- Beware of workbench **state**. The tests within a single suite will share the same state.
2017-05-22 11:56:23 +00:00
2017-09-25 20:50:39 +00:00
- Beware of **singletons**. This evil can, and will, manifest itself under the form of FS paths, TCP ports, IPC handles. Whenever writing a test, or setting up more smoke test architecture, make sure it can run simultaneously with any other tests and even itself. All test suites should be able to run many times in parallel.
2017-09-04 09:32:41 +00:00
2017-09-25 20:50:39 +00:00
- Beware of **focus**. **Never** depend on DOM elements having focus using `.focused` classes or `:focus` pseudo-classes, since they will lose that state as soon as another window appears on top of the running VS Code window. A safe approach which avoids this problem is to use the `waitForActiveElement` API. Many tests use this whenever they need to wait for a specific element to _have focus_.
2017-09-25 20:46:02 +00:00
- Beware of **timing**. You need to read from or write to the DOM... but is it the right time to do that? Can you 100% guarantee that `input` box will be visible at that point in time? Or are you just hoping that it will be so? Hope is your worst enemy in UI tests. Example: just because you triggered Quick Open with `F1`, it doesn't mean that it's open and you can just start typing; you must first wait for the input element to be in the DOM as well as be the current active element.
2017-10-10 12:41:11 +00:00
2018-04-18 10:01:17 +00:00
- Beware of **waiting**. **Never** wait longer than a couple of seconds for anything, unless it's justified. Think of it as a human using Code. Would a human take 10 minutes to run through the Search viewlet smoke test? Then, the computer should even be faster. **Don't** use `setTimeout` just because. Think about what you should wait for in the DOM to be ready and wait for that instead.