Update 17-01-2016: Added Protractor: Tips for better End-to-End tests
In my previous blog post I mentioned that our team had recently started building an AngularJS application. Well we are a few more weeks further along and we’ve spent the last few weeks overcoming some initial challenges such as using spies, injecting services and debugging the tests.
When we started developing our AngularJS app during my research into testing options I came across Protractor:
“Protractor runs tests against your application running in a real browser, interacting with it as a user would.”
In this post i will talk through our decision process for automating, why we chose protractor and how we initially got started.
Why automate through front end?
At this point i think it’s important to share our decision process for choosing why and what to automate. As a team we have used automated E2E tests using WebDriver in the past for one of our other products, but we didn’t really get much return for our effort. Our main product is quite hard to automate and at the time we weren’t really in a great position to automate. Since building our AngularJS app we wanted to revisit our E2E automation. Thankfully this time round we had the following in place:
- Good foundations through Unit Tests and Integration Tests
- Testable application through sensible element naming and markup
- At least one E2E journey developed
- Critical user journeys to be automated are agreed with team
This meant we were in a much better starting position. We were also clear on our objective for the automated front end tests:
“After each deployment ensure our critical user flows continue work E2E”
We initially identified 4 user journeys. We also made the decision that our E2E automation would not be part of every user story. Instead we would automate at a “Feature” level for complete user flows. This doesn’t mean we don’t automate on each story, we still of course do lots of unit and integration tests which are backed up by lots of exploratory testing.
This article by Brian Marick also gives a good approach on deciding what to automate and when.
So now you know a bit more about Why, What and When lets go back to Protractor….
Why use Protractor rather than WebDriver out of the box?
So most people have probably heard of WebDriver. You can automate tests in a variety of different languages and we could have automated our AngularJS app without Protractor. One of the main draws of Protractor for our team was its synchronization which can be one of the challenges when automating websites.
“You no longer need to add waits and sleeps to your test. Protractor can automatically execute the next step in your test the moment the webpage finishes pending tasks, so you don’t have to worry about waiting for your test and webpage to sync.”
Thankfully it’s a pretty similar process to other NodeJS stuff:
- Install packages
- Do your Config
- Run through NodeJS command prompt
The Protractor site has a good guide on how to do the basic setup.
Page Object pattern
Design of automated tests is really important and being familiar with the Page Object Pattern we wanted to follow a similar approach for our Protractor tests. Thankfully we came across this excellent article.
Another important part of any automated testing is handling your data. Thankfully you can do this quite easily with protractor. You can declare your test data in your protractor config and then read it into your test. Here’s how to do it.
Parallel running of tests
Speed of tests is also important to give the team quick feedback. Although we only have 4 user journeys automated we wanted to get parallel running of the tests in early. Again Protractor supports this out of the box. Here is how you do it.
With this and the synchronization benefit of protractor mentioned earlier we find the tests run super quick. To run our 4 scenarios in Firefox it takes around 3 minutes.
Knowing what’s passed and more importantly failed is of course a must. We have used the Protractor-HTML-Screenshot-reporter for this. It gives you a basic report template showing your tests and a screenshot if the tests fail.
The final part of getting everything up and running was to create a Jenkins job to run our tests. This was again super easy. Just install everything you need on your Jenkins server and push the relevant configuration files. Then all you do is add a command to the job to run protractor. Our tests run after every deployment to our test environment and we also display our reports within Jenkins for each test run. Super.
So far the tests are giving us some good feedback. A big benefit we have found with protractor is the synchronization. We haven’t got any thread.sleeps or waits within our test code and so far we haven’t had any synchronization problems which has meant a lot less false negatives. Our next steps are to abstract our data out further in JSON files, change our tests so they run in a specific firefox profile we control and also introducing Internet Explorer. At this point we probably won’t expand the number of scenarios right away but instead invest time to improving the capabilities of the test code and making it more robust.