You cannot Automate a User Journey Test

Standard

Most Test Automation models describe User Journey tests as an important component. In the case of the most well loved model, the Test Automation Pyramid, you’ll usually find User Journey tests towards the top(as per below): B95IkAFIQAAxr2z

These are typically some form of end-to-end journey, usually automated through the User Interface, using tools such as Selenium WebDriver, QTP etc.

Why can’t you automate a User Journey Test?

Lets start by defining a User Journey:

User journeys tap directly reflect the thoughts, considerations, and experiences that people go through in their daily lives, beyond the web. Creating a user journey places a strong emphasis on personas and also merges the creation of scenarios and user flows. However, unlike user flows, hierarchies, or functional specs (which explain the interaction between a user and a system’s logic and processes), user journeys explore a user’s mental and lived “patterns, processes, and paths” and translate these into web-based experiences.” [1]

user_journey

The important part of the definition is that a User Journey is a “mental” and “lived” experience. The journey is deeply linked to “emotions” and these emotions usually have a bearing on the users perception of quality. Within development teams most of our work is focused either on enhancing existing user journeys or creating completely brand new journeys with the work typically broken down into “User Stories”. When building user journeys the success to which we have met the users is needs is based upon:

  • Have we me the users goals
  • Have we understood their motivations
  • Have we alleviated some of their existing pain points
  • Have delivered something that matches their character
  • Does it enable the user to do what needs to be done

As delivering a successful User Journey is so important to many teams its no surprise that many teams look to maximise their testing efforts in this area with some looking to automation. However, as described the key success factors in a user journey are linked to Human Emotion. As it stands, these emotions cannot be encoded into an Automated Test(although in the future it might be possible[2])

Due to this we cannot automate a User Journey test. kobian_sadness_1429339c

Testing the emotions

As a tester some of the most valuable feedback we can provide is on an emotional level.

“Emotions and Feelings are signals. Look into what they’re signalling” [3] 

It is only through experiencing the journey on an emotional level, as human testers, that can we provide feedback on the ‘journey’. A difficult part of testing is to really understand deeply the emotions of people that matter.

– Who is the user in the journey?

– What is their character?

– What are their motivations?

– What are their pain points?

How well we understand the emotions of our users can be an important factor in the success of our testing. If we do not really understand our users on an emotional level we may overlook certain issues such as clunky navigation, slow page responses or misleading messaging.

A great tool we can use(and not, its not Selenium Webdriver) is Personas. These are a great way of “getting inside” the mind of the user. Most agile teams will be familiar with personas as they form part of most User Story templates:

“As Stewart i want to be informed when my loan is due to expire so that i can avoid hefty late return fees”

In my current team we spent alot of time breathing life into our Personas giving them interesting names, personal backgrounds, job titles, goals and objectives. One of our personas is “Sarah the System Admin”. These personas reflect our real user base. We also had “face time” with the users understanding their emotions when using our software. One of our teams even created baseball cards of their personas so it was even easier for them to remember all the ‘players’.

At BDD Exchange 2014 Jeff Patton encouraged the attendees to “Get out of the building, way out of the building” to see how real users interact with our software. A better understanding of user emotions can breathe life into what may be seen as mundane tests. It also allows issues to be discussed on an emotional level “I’m pretty sure Stewart wouldn’t be happy with this flow, he said he wanted something quick, if possible, 1 click” Unfortunately too often this form of ’emotional’ testing falls into a Usability Testing phase. In traditional Usability Testing we prepare a prototype to show the users, usually in an expensive ‘off site’ test center. Following a couple of feedback iterations the idea moves into a lengthy development cycle and once complete, we schedule another session to take the users through the ‘final’ solution. We then go through the development iteration again after we realise further changes are required. Often the gap between this feedback is too long for most teams.

As Testers we can infuse the whole development cycle with our user persona’s. We can ensure their emotions and feelings are considered l so that the developed solution is likely to be closer to the users ’emotional expectations’. As Testers we can do some of our best testing before a line of code has been written through testing paper prototypes, whiteboard designs and challenging assumptions around user expectations.

Many organizations are now realizing the benefits of this type of testing throughout the entire development life cycle. My previous company, Shop Direct Group, are somewhat trendsetters in this area. They have built an in-house customer testing lab located in the center of the office, bringing the customer to the heart of the development process, literally! [4]

Design Thinking [5] is also an emerging as a popular approach. In this the emotion of ’empathy’ is the start of the process.

design_think

If you are interested in understanding more about how our emotions influence us in testing, Stephen Janaway has a great slide deck exploring the topic further [6]

In all these examples human emotions are an integral part and Automated tests cannot provide this feedback.

So what can be automated?

Implicit to a good Customer Journey is at least an application that holds together. If i expect a 5 and i get a 2, or if the user expects the home page to be displayed after Login but they get a 404 Page, we can probably assume its not going to be a great Customer Journey. Automation can help in ‘checking’ our application is at least doing what we expect it to. James Bach differentiates these types of checks from testing:

“Checks are evaluations by applying algorithmic decision rules to specific observations of a product.” [7]

Although we cannot automate a User Journey Test to evaluate the emotional aspects it doesn’t mean that we can forget the user in our automated checks. Understanding the ‘flows’ which different user segments take through the system is vital. Observing users on your application and using data analytic tools from vendors such as Google can build a picture of the most popular flows. We can then create automated checks to ensure that the flow between these pages is at least working. In my experience teams can often neglect the user in their checks. Many automated checks take the ‘path of least resistance’. Rather than automating the most popular user journeys the checks are designed to minimize automation maintenance.

For example, at a previous company we automated quite heavily through the User Interface. A key part of this was adding items to basket and ordering. Our automation checks would start by going directly to a product, we would pass in the Product ID as follows:

http://www.worldbestshop.com/?productId=ItemNumberGoesHere

This was efficient for automation purposes but meant we were skipping one of the most important parts of the customer flow the home page, search and gallery pages, which were the most visited pages of the site.

Our automation also needs to consider user ‘Persona’s’ from a data perspective. Do the users in our automated tests reflect what a real user looks like? Again, in my experience, teams often bulk load in a set of users who are all pretty much exactly the same, again to minimize the overhead of managing lots of different data types. Therefore you’ll often miss issues that occur in the differences between users. A good example is a user returning to a shopping site when they have already added an item to basket. Do all your test start with an empty basket?

To Conclude….

A User Journey is an ’emotional’ experience so we cannot automate these type of tests yet(and probably never will). Ensuring you consider the emotions of your users throughout the entire development life cycle can increase your chances of having an application that will create a good experience for your users. As Testers we can play a valuable part in testing with our users emotions at heart.

Automation can provide an important platform for any customer journey to ensure the application is at least giving the customer the correct result from business logic and allowing to flow through the system.

[1] Introduction to User Journeyshttp://boxesandarrows.com/an-introduction-to-user-journeys/

[2] Softbank unveils ‘human-like’ robot Pepper – http://www.bbc.co.uk/news/technology-27709828

[3] Emotions in Software Testinghttp://www.developsense.com/presentations/2013-05-STAREast-EmotionsInSoftwareTesting.pdf

[4] Shop Direct Test Labhttp://www.shopdirect.com/shop-direct-accelerates-testing-programme-in-house-ux-lab/

[5] Testing your Emotions – http://stephenjanaway.co.uk/stephenjanaway/wp-content/uploads/2014/06/Testing-Your-Emotions.pdf

[6] Design Thinking – http://designthinking.ideo.com/

[7] Testing and Checking refinedhttp://www.satisfice.com/blog/archives/856

Advertisements

One thought on “You cannot Automate a User Journey Test

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s