Smoke Testing With Puppeteer & Jasmine
Here at Innovid, a leading video marketing platform where we serve 1.3 million hours of video per day, we must make sure that all of our UI applications are available after each change.
Continuous Deployment Process
First, let’s describe our continuous deployment architecture and infrastructure. We use Bitbucket as our web-based version control repository hosting service and TeamCity as our build management and continuous integration server. Every change that is being pushed to Bitbucket (via Git) causes a chain of builds (jobs or tasks) to run on TeamCity. The chain includes various white-box and black-box tests which results in a package or image creation of the new changed application. The package or image creation eventually results in deployment builds which deploy the application to our staging and then production environments.
Monitoring
We monitor our applications periodically with two tools: Sensu for error monitoring and Pingdom for health checking. These tools helped us identify production incidents in the past and have proven themselves as great, enabling us to respond as fast as we can to any issue that exists in our live production environment. We use Opsgenie to manage and assign alerts from these tools to developers or Tier2 engineers.
Motivation
The above process worked great until we had the first UI incident in production. Users complained that they cannot reach one of our dashboards. Our Tier2 engineers verified and reported that the dashboard always redirects to a “Not Found” page. How is this possible? All white-box and black-box tests were fine and the application worked flawlessly on our staging environment. Furthermore, no errors or health issues were found by the monitoring tools. A group from our skilled R&D team identified the root cause and fixed it ASAP. The issue existed for a couple of hours but left some minds full of thoughts.
What was missing?
We did almost everything right. We had a continuous deployment process without user intervention in place. The module was covered with hundreds of tests including E2E and UI tests, while monitoring tools checked that the application is live without any error in production.
What we didn’t have is someone that will check that the UI is actually available for users in production after the deployment of new code. But do we really want developers to check that the application is live in production after every development change? Dozens of changes can happen every day and mistakes may happen.
Steve McConnell used the term: “switch it on and see if smoke comes out” in his book “Rapid Development” and we took this simple statement and built an automated tool that will do it for us.
Post Deployment Smoke Testing
The vision that we set to achieve is a tool or library that would help us identify if an application is live and available to users (humans and not monitoring tools). We built a NodeJS library called Hygieia which has its own TeamCity build chain that publishes the library to our NPM repository.
Hygieia is a wrapper that connects tests written with Jasmine to Puppeteer which exposes an API for interactions with a headless Chrome browser. Hygieia also connects Jasmine test runner to a TeamCity reporter in order to have the test report inside TeamCity’s build log. The library has some extra internal functionality such as company cookie generation/injection etc. We usually don’t like wrappers, but the goal is to make a simple “plug and play” library that, again, will check if “smoke comes out”.
After each deployment, as the last step of the deployment (staging and production), we run the Hygieia smoke test. if it fails then we can roll back and stop the continuous deployment chain to production. In those cases the build in TeamCity will be marked as “failed” which points to an issue in the last attempted code change deployment.
The setup:
The smoke test (a.k.a spec file):
The run command:
Thats it. Simple as that.
Conclusions
- You can use Puppeteer and Jasmine without wrapping them in a library if your use case is as simple as described above. Our case was a bit more complicated and the library includes various features such as cookie injection, statsd metrics, Flash engine support, SSO login, retry mechanism, etc.
- Developers tend to develop unnecessary UI testing with those tools instead of simply “switching it on and seeing if smoke comes out”. In this case, “switching it on” means “navigating to the live website” and “seeing if smoke comes out” is “checking some usual element/s on that website”. If the usual element doesn’t exist then “smoke comes out!!!“. The rule of thumb is that the test should include around 20 lines of code.
- Make sure to stop the deployment chain or roll back when the test fails.
- It’s OK to make some mistakes, it’s not OK to do nothing about them.
