![]() * Add a bunch of tests * tweak tests * add profile tests, now breaking (no profiles showing up) * uncomment tests * update expected * fix test * merge * all passing * bunch of improvements * improve readability * all should be passing and cleaned up * all passing * fix #137243 and clean-up * remove .only * fix #137247 * get rid of enum * see if this fixes timing issue * use keybinding for show terminal instead * remove delay, implement better accept function * get rid of unneeded index * try something else * wait for empty editor to be active * remove .only * decrease redundancy in names * re-add xterm selector * try something else * undo change that broke things worse * add terminal-tabs test * all tabs tests passing locally * kill all * await before running each * fix tests * remove .only * try waiting for focused xterm * remove .only * clean up * 🧹 * fix profiles tests by using this.app.props * revert to use default shell * re-add conditional * Get tests passing on WebKit On WebKit the smoke tests were failing because the quick pick was triggering a focus event on the terminal after the blur event had already happened, this caused the view service to think the terminal was still focused when it wasn't. The fix was to reset the context key also in IViewsService.closeView. * Safari -> WebKit * Prefer arrow functions * remove beforeEach timeout * add timeouts to contributed profile tests * add terminal editor tests, fix #137377 * add a bunch of tests * refactor * get rid of profile specific command * more polish * refactor getTabs -> assertTerminalGroups * fix failing test * more polish * remove .only * add assertSingleTab * remove unused import * fix error * fix more failures * more changes * fix almost all except for plus button * fix failing plus button test * all passing * modify error that gets thrown * modify error message again * remove unused wildcard/ ANY_NAME, fix icon code * fix icon code * large refactor, improvements * finish polish * 1 failing * get test to pass Co-authored-by: Daniel Imms <2193314+Tyriar@users.noreply.github.com> |
||
---|---|---|
.. | ||
src | ||
test | ||
.gitignore | ||
Audit.md | ||
package.json | ||
README.md | ||
tsconfig.json | ||
yarn.lock |
VS Code Smoke Test
Make sure you are on Node v12.x.
Quick Overview
# Build extensions in the VS Code repo (if needed)
yarn && yarn compile
# Install Dependencies and Compile
yarn --cwd test/smoke
# Prepare OSS in repo*
node build/lib/preLaunch.js
# Dev (Electron)
yarn smoketest
# Dev (Web - Must be run on distro)
yarn smoketest --web --browser [chromium|webkit]
# Build (Electron)
yarn smoketest --build <path to latest version>
example: yarn smoketest --build /Applications/Visual\ Studio\ Code\ -\ Insiders.app
# Build (Web - read instructions below)
yarn smoketest --build <path to server web build (ends in -web)> --web --browser [chromium|webkit]
# Remote (Electron - Must be run on distro)
yarn smoketest --build <path to latest version> --remote
* This step is necessary only when running without --build
and OSS doesn't already exist in the .build/electron
directory.
Running for a release (Endgame)
You must always run the smoketest version that matches the release you are testing. So, if you want to run the smoketest for a release build (e.g. release/1.22
), you need to check out that version of the smoke tests too:
git fetch
git checkout release/1.22
yarn && yarn compile
yarn --cwd test/smoke
Web
There is no support for testing an old version to a new one yet.
Instead, simply configure the --build
command line argument to point to the absolute path of the extracted server web build folder (e.g. <rest of path here>/vscode-server-darwin-web
for macOS). The server web build is available from the builds page (see previous subsection).
macOS: if you have downloaded the server with web bits, make sure to run the following command before unzipping it to avoid security issues on startup:
xattr -d com.apple.quarantine <path to server with web folder zip>
Note: make sure to point to the server that includes the client bits!
Debug
--verbose
logs all the low level driver calls made to Code;-f PATTERN
(alias-g PATTERN
) filters the tests to be run. You can also use pretty much any mocha argument;--screenshots SCREENSHOT_DIR
captures screenshots when tests fail.--headless
will run playwright in headless mode when--web
is used.
Note: you can enable verbose logging of playwright library by setting a DEBUG
environment variable before running the tests (https://playwright.dev/docs/debug#verbose-api-logs)
Develop
cd test/smoke
yarn watch
Troubleshooting
Error: Could not get a unique tmp filename, max tries reached
On Windows, check for the folder C:\Users\<username>\AppData\Local\Temp\t
. If this folder exists, the tmp
module can't run properly, resulting in the error above. In this case, delete the t
folder.
Pitfalls
-
Beware of workbench state. The tests within a single suite will share the same state.
-
Beware of singletons. This evil can, and will, manifest itself under the form of FS paths, TCP ports, IPC handles. Whenever writing a test, or setting up more smoke test architecture, make sure it can run simultaneously with any other tests and even itself. All test suites should be able to run many times in parallel.
-
Beware of focus. Never depend on DOM elements having focus using
.focused
classes or:focus
pseudo-classes, since they will lose that state as soon as another window appears on top of the running VS Code window. A safe approach which avoids this problem is to use thewaitForActiveElement
API. Many tests use this whenever they need to wait for a specific element to have focus. -
Beware of timing. You need to read from or write to the DOM... but is it the right time to do that? Can you 100% guarantee that
input
box will be visible at that point in time? Or are you just hoping that it will be so? Hope is your worst enemy in UI tests. Example: just because you triggered Quick Access withF1
, it doesn't mean that it's open and you can just start typing; you must first wait for the input element to be in the DOM as well as be the current active element. -
Beware of waiting. Never wait longer than a couple of seconds for anything, unless it's justified. Think of it as a human using Code. Would a human take 10 minutes to run through the Search viewlet smoke test? Then, the computer should even be faster. Don't use
setTimeout
just because. Think about what you should wait for in the DOM to be ready and wait for that instead.