Nx Console is the UI for Nx & Lerna. It’s available as a VSCode extension (and more IDEs coming soon!) and with it, you get powerful features like:
autocomplete for Nx config files,
exploration of generators (and schematics) in a form-based view,
context-aware visualisation of the Nx dependency graph right in your IDE
and many more DX improvements throughout the extension!
The Nx Console project started many years ago and went through various iterations. From a small, standalone client for angular schematics to what it is today: A fully-integrated tool to help you be more productive while developing with Nx.
And folks like it: We passed a million installs this year! 🎉
Of course we as maintainers need to make sure that all of our features work when we make a PR or release a new version.
This used to mean manually clicking through features in different sample workspaces - as you can imagine, this quickly became tedious as well as unreliable. Keep in mind that Nx Console works not just for Nx but also for Lerna and Angular workspaces across multiple versions!
We needed a solution to automate this.
The Problem with testing VSCode extensions
There’s many different kinds of tests and different ways to structure and write them. Let’s go over four common types of tests and see how to use them in the context of a VSCode extension.
Static Tests, like type checking and linting, are a given. VSCode extensions are written in Typescript and Nx sets up ESLint for us automatically so we get these static checks for free.
Unit Tests break your code down into ‘units’ and test those in isolation (what exactly constitutes a ‘unit’ is up to interpretation in most cases). They are a bit more complicated to get right here. Because a lot of functionality is tied to the VSCode APIs, unit tests often end up mocking everything which results in little functionality actually being tested.
Because of this, unit tests are mostly useful for niches as well as helper and utility functions, not for broad coverage. They can still be useful and we have unit tests written with jest throughout our repo (but you could use any JS test runner to write and run unit tests).
Integration Tests combine multiple parts of your software and test them together. They are a good option for testing some behaviour of extensions. If you read the docs, they suggest using the @vscode/test-electron package and mocha. This will allow you to run tests inside an actual VSCode instance, so you avoid mocking. However, you are still constrained. The API gives limited information on many areas. For example even this very simple test is not easily realizable with the VSCode API:
suite("Sample Extension",()=>{vscode.commands.executeCommand("testing-ext.helloWorld");test("should display 'Hello World' notification",()=>{// ??? - VSCode API does not expose this information});});
End-to-End (E2E) tests are robots that click through your application to make sure it works. They can test every kind of behaviour an actual user could do. Of course, this comes with a price: The overhead to run them can be quite high as you need to spin up fresh VSCode instances for each test. But this is a tradeoff worth taking: You will not be restricted in what you test and the tests mimic actual user flows. The best part: Since VSCode is built on Electron, there is a DOM that you can read, manipulate and assert on just like you would with any web application and keep using the well-established JS tooling we have for this. This is where WebdriverIO comes into play.
WebdriverIO and the wdio-vscode-service
WebdriverIO (abbreviated as WDIO) is an E2E testing framework for Node.js. It allows you to automate all kinds of web and mobile applications using the Webdriver or Chrome DevTools protocols.
In theory - since VSCode is built on Electron and is just Javascript, HTML and CSS under the hood - you could use any other framework like Cypress or Nightwatch to test it.
So why WebdriverIO?
WDIO has one big advantage: The wdio-vscode-service . It’s a plugin that integrates with WDIO and handles all setup-related work for you. You won’t have to think about downloading, installing and running a VSCode instance or the matching Chromedriver. On top of that, it bootstraps page objects for many functionalities and lets you access the VSCode API via remote function execution.
All these (and more) features enable you to set WDIO up quickly and move on to writing test!
Configuring WDIO and writing the first test
Setting up WDIO is a straightforward process. By following the installation guide you get a preconfigured wdio.conf.ts file and all required dependencies installed. After adding some types to tsconfig.json and tweaking some paths in wdio.conf.ts to match our folder structure, we were already at the point where we could execute wdio run ./wdio.conf.ts . You can see that VSCode is downloaded and unpacked for us. But of course, without any tests nothing really happens. Let’s change that!
➜ vscode-e2e ✗ npx wdio run ./wdio.conf.ts
Execution of 0 workers started at 2022-10-27T21:39:54.555Z
Downloading VS Code 1.72.2 from https://update.code.visualstudio.com/1.72.2/darwin/stable
Downloading VS Code [==============================] 100%
Downloaded VS Code into /Users/maxkless/nx-console/apps/vscode-e2e/.wdio-vscode-service/vscode-darwin-1.72.2
2022-10-27T21:40:42.539Z ERROR @wdio/cli:launcher: No specs found to run, exiting with failure
Spec Files: 0 passed, 0 total (0% completed)in 00:00:47
Writing a test isn’t very complicated. Create a file in a location matching your configuration’s specs property and WDIO will pick it up automatically. You can use mocha, cucumber or jasmine as test frameworks and start writing describe , it and before blocks like you’re used to.
import{Workbench}from'wdio-vscode-service';describe('NxConsole Example',()=>{it('should be able to load VSCode',async ()=>{constworkbench:Workbench=awaitbrowser.getWorkbench();expect(awaitworkbench.getTitleBar().getTitle()).toBe('[Extension Development Host] Visual Studio Code');});});
Refer to the WebdriverIO docs to learn more about its API and how to write tests.
If we run wdio run ./wdio.conf.ts again, we will see that WDIO is using the cached VSCode binary and successfully executing our test!
vscode-e2e ✗ npx wdio run ./wdio.conf.ts
Execution of 1 workers started at 2022-11-10T12:10:40.029Z
Found existing install in /Users/maxkless/nx-console/apps/vscode-e2e/.wdio-vscode-service/vscode-darwin-1.72.2. Skipping download
[0-0] RUNNING in chrome - /specs/example.e2e.ts
[0-0] PASSED in chrome - /specs/example.e2e.ts
"spec" Reporter:
------------------------------------------------------------------[chrome 102.0.5005.167 mac os x #0-0] Running: chrome (v102.0.5005.167) on mac os x[chrome 102.0.5005.167 mac os x #0-0] Session ID: da14f0f07ba2fe47728eaf8a249b5bca[chrome 102.0.5005.167 mac os x #0-0][chrome 102.0.5005.167 mac os x #0-0] » /specs/example.e2e.ts[chrome 102.0.5005.167 mac os x #0-0] NxConsole Example[chrome 102.0.5005.167 mac os x #0-0] ✓ should be able to load VSCode[chrome 102.0.5005.167 mac os x #0-0][chrome 102.0.5005.167 mac os x #0-0] 1 passing (1.3s)
Spec Files: 1 passed, 1 total (100% completed)in 00:00:28
Integrating with Nx - defining a target
Of course, we wanted to take advantage of Nx’s powerful features like the task graph, computation caching and distributed task execution. If you’re working on an integrated style Nx repo, you get access to many official and community plugins that allow for instant integration with popular dev tools. Jest, ESLint, Cypress and many more have executors (more on that here) that allow you to run them through Nx. This isn’t the case for WebdriverIO, so we had two options: Create a custom plugin and WDIO executor or simply use nx:run-commands to wrap arbitrary commands with Nx. If WDIO became widely used in our repo, writing a custom plugin isn’t too much effort and could definitely be worth it! But for this one-time usage, we went with the quicker option. Let’s set up an e2e target like this:
{"$schema":"../../node_modules/nx/schemas/project-schema.json","sourceRoot":"apps/vscode-e2e/src","projectType":"application","targets":{"e2e":{"executor":"nx:run-commands","options":{"command":"wdio run ./wdio.conf.ts","cwd":"apps/vscode-e2e"},"dependsOn":["^build"]}},"implicitDependencies":["vscode","nxls"],}
Now, if we run nx run vscode-e2e:e2e , we will see WDIO run inside Nx!
➜ nx-console ✗ npx nx run vscode-e2e:e2e
✔ 2/2 dependent project tasks succeeded [2 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run vscode-e2e:e2e
## Execution of 1 workers started at 2022-10-27T19:44:02.756Z
Found existing install in /Users/maxkless/nx-console/apps/vscode-e2e/.wdio-vscode-service/vscode-darwin-1.71.2. Skipping download
[0-0] RUNNING in chrome - /specs/example.e2e.ts
[0-0] PASSED in chrome - /specs/example.e2e.ts
"spec" Reporter:
[chrome 102.0.5005.167 mac os x #0-0] Running: chrome (v102.0.5005.167) on mac os x[chrome 102.0.5005.167 mac os x #0-0] Session ID: 74edae2b6a5f68effc8bdec1e2114e5f[chrome 102.0.5005.167 mac os x #0-0][chrome 102.0.5005.167 mac os x #0-0] » /specs/example.e2e.ts[chrome 102.0.5005.167 mac os x #0-0] NxConsole Example[chrome 102.0.5005.167 mac os x #0-0] ✓ should be able to load VSCode[chrome 102.0.5005.167 mac os x #0-0][chrome 102.0.5005.167 mac os x #0-0] 1 passing (6.8s)
Spec Files: 1 passed, 1 total (100% completed)in 00:00:29
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target e2e for project vscode-e2e and 2 task(s) it depends on (40s)>
Nx read the output from the cache instead of running the command for 2 out of 3 tasks.
See Nx Cloud run details at [https://cloud.nx.app/runs/rprfQeMdy6](https://cloud.nx.app/runs/rprfQeMdy6)
While the output doesn’t look too different, this enables some amazing features! If we run the tests again, they will complete in a few milliseconds because the result could be retrieved from cache. Also, because we defined dependsOn and implicitDependencies , Nx will always make sure that up-to-date versions of Nx Console and the nxls are built before E2E tests run. All this with just a few lines of config!
Setting up CI
Another important step in getting the maximum value out of automated E2E testing is running it in CI. Adding new checks to an existing Github Actions pipeline isn’t complicated, so a single line of yaml configuration should do the trick:
nx affected analyzes your code changes in order to compute the minimal set of projects that need to be retested. Learn more about it here: How Affected Works
This will fail, however, because WebdriverIO tries to open VSCode and expects a screen - which action runners obviously don’t have. If we were testing on a simple Chrome or Firefox instance, this could be solved by adding --headless to the browser’s launch options. VSCode doesn’t support a headless mode, though, so we had to find another solution: xvfb.
xvfb, short for X virtual frame buffer, is a display server that allows you to run any program headlessly by creating a virtual display - the frame buffer. To run our E2E test through xvfb on CI, two steps were necessary:
First, we created a new configuration in our e2e target that runs the same script, but through xvfb.
{..."e2e":{..."configurations":{"ci":{"command":"xvfb-run -a wdio run ./wdio.conf.ts"}},}},}
Second, we have to make sure xvfb is installed on our action runners. Theoretically, we could just run sudo apt-get install -y xvfb on all our runners and call it a day. But for clarity’s sake and to demonstrate some of the advanced things you can do with Nx Cloud, we decided on a different solution. We want to create two kinds of runners: One with xvfb installed and one without it. This can be done by using the NX_RUN_GROUP variable in our agent and task definitions. With it, E2E tests are run on the first kind and other tasks are run on any runner.
First, we specify a unique value of NX_RUN_GROUP per run attempt and set it as an environment variable in our agent definition. Then, we make sure xvfb is installed on these agents.
Nx Cloud then matches these run groups and makes sure that all E2E tasks are only executed on agents with xvfb installed. With it, we can now do everything we can do locally, with a real screen. For example, we take screenshots on failure and make them available to download from GitHub via actions/upload-artifact .
Conclusion
In the end, we have fully functioning E2E test running on every PR, fully cached and distributable through Nx.
I want to take a moment and give special thanks to Christian Bromann who is a lead maintainer on WebdriverIO and the wdio-vscode-service. Without his foundational work, this would’ve been 100x harder.
If you have questions, comments or just want to chat about Nx Console, you can find me on twitter: @MaxKless
Some of the code snippets I showed here have been shortened for clarity, but if you want to see our setup with all details, head over to GitHub:
Spend less time looking up command line arguments and more time shipping incredible products.
Why Nx Console?
Developers use both command-line tools and user interfaces. They commit in the terminal, but resolve conflicts in Visual
Studio Code or WebStorm. They use the right tool for the job.
Nx is a command-line tool, which works great when you want to serve an application or generate a simple component. But
it falls short once you start doing advanced things.
For instance:
Exploring custom generator collections is hard in the terminal, but it's easy using Nx Console.
Using rarely-used flags is challenging. Do you pass absolute or relative paths? You don't have to remember any flags
names or paths - Nx Console will help you by providing autocompletion and validating your inputs.
Context-switching between your IDE and the browser is annoying. With Nx Console, you can view…
MaxKless
