You cannot Automate a User Journey Test


Most Test Automation models describe User Journey tests as an important component. In the case of the most well loved model, the Test Automation Pyramid, you’ll usually find User Journey tests towards the top(as per below): B95IkAFIQAAxr2z

These are typically some form of end-to-end journey, usually automated through the User Interface, using tools such as Selenium WebDriver, QTP etc.

Why can’t you automate a User Journey Test?

Lets start by defining a User Journey:

User journeys tap directly reflect the thoughts, considerations, and experiences that people go through in their daily lives, beyond the web. Creating a user journey places a strong emphasis on personas and also merges the creation of scenarios and user flows. However, unlike user flows, hierarchies, or functional specs (which explain the interaction between a user and a system’s logic and processes), user journeys explore a user’s mental and lived “patterns, processes, and paths” and translate these into web-based experiences.” [1]


The important part of the definition is that a User Journey is a “mental” and “lived” experience. The journey is deeply linked to “emotions” and these emotions usually have a bearing on the users perception of quality. Within development teams most of our work is focused either on enhancing existing user journeys or creating completely brand new journeys with the work typically broken down into “User Stories”. When building user journeys the success to which we have met the users is needs is based upon:

  • Have we me the users goals
  • Have we understood their motivations
  • Have we alleviated some of their existing pain points
  • Have delivered something that matches their character
  • Does it enable the user to do what needs to be done

As delivering a successful User Journey is so important to many teams its no surprise that many teams look to maximise their testing efforts in this area with some looking to automation. However, as described the key success factors in a user journey are linked to Human Emotion. As it stands, these emotions cannot be encoded into an Automated Test(although in the future it might be possible[2])

Due to this we cannot automate a User Journey test. kobian_sadness_1429339c

Testing the emotions

As a tester some of the most valuable feedback we can provide is on an emotional level.

“Emotions and Feelings are signals. Look into what they’re signalling” [3] 

It is only through experiencing the journey on an emotional level, as human testers, that can we provide feedback on the ‘journey’. A difficult part of testing is to really understand deeply the emotions of people that matter.

– Who is the user in the journey?

– What is their character?

– What are their motivations?

– What are their pain points?

How well we understand the emotions of our users can be an important factor in the success of our testing. If we do not really understand our users on an emotional level we may overlook certain issues such as clunky navigation, slow page responses or misleading messaging.

A great tool we can use(and not, its not Selenium Webdriver) is Personas. These are a great way of “getting inside” the mind of the user. Most agile teams will be familiar with personas as they form part of most User Story templates:

“As Stewart i want to be informed when my loan is due to expire so that i can avoid hefty late return fees”

In my current team we spent alot of time breathing life into our Personas giving them interesting names, personal backgrounds, job titles, goals and objectives. One of our personas is “Sarah the System Admin”. These personas reflect our real user base. We also had “face time” with the users understanding their emotions when using our software. One of our teams even created baseball cards of their personas so it was even easier for them to remember all the ‘players’.

At BDD Exchange 2014 Jeff Patton encouraged the attendees to “Get out of the building, way out of the building” to see how real users interact with our software. A better understanding of user emotions can breathe life into what may be seen as mundane tests. It also allows issues to be discussed on an emotional level “I’m pretty sure Stewart wouldn’t be happy with this flow, he said he wanted something quick, if possible, 1 click” Unfortunately too often this form of ’emotional’ testing falls into a Usability Testing phase. In traditional Usability Testing we prepare a prototype to show the users, usually in an expensive ‘off site’ test center. Following a couple of feedback iterations the idea moves into a lengthy development cycle and once complete, we schedule another session to take the users through the ‘final’ solution. We then go through the development iteration again after we realise further changes are required. Often the gap between this feedback is too long for most teams.

As Testers we can infuse the whole development cycle with our user persona’s. We can ensure their emotions and feelings are considered l so that the developed solution is likely to be closer to the users ’emotional expectations’. As Testers we can do some of our best testing before a line of code has been written through testing paper prototypes, whiteboard designs and challenging assumptions around user expectations.

Many organizations are now realizing the benefits of this type of testing throughout the entire development life cycle. My previous company, Shop Direct Group, are somewhat trendsetters in this area. They have built an in-house customer testing lab located in the center of the office, bringing the customer to the heart of the development process, literally! [4]

Design Thinking [5] is also an emerging as a popular approach. In this the emotion of ’empathy’ is the start of the process.


If you are interested in understanding more about how our emotions influence us in testing, Stephen Janaway has a great slide deck exploring the topic further [6]

In all these examples human emotions are an integral part and Automated tests cannot provide this feedback.

So what can be automated?

Implicit to a good Customer Journey is at least an application that holds together. If i expect a 5 and i get a 2, or if the user expects the home page to be displayed after Login but they get a 404 Page, we can probably assume its not going to be a great Customer Journey. Automation can help in ‘checking’ our application is at least doing what we expect it to. James Bach differentiates these types of checks from testing:

“Checks are evaluations by applying algorithmic decision rules to specific observations of a product.” [7]

Although we cannot automate a User Journey Test to evaluate the emotional aspects it doesn’t mean that we can forget the user in our automated checks. Understanding the ‘flows’ which different user segments take through the system is vital. Observing users on your application and using data analytic tools from vendors such as Google can build a picture of the most popular flows. We can then create automated checks to ensure that the flow between these pages is at least working. In my experience teams can often neglect the user in their checks. Many automated checks take the ‘path of least resistance’. Rather than automating the most popular user journeys the checks are designed to minimize automation maintenance.

For example, at a previous company we automated quite heavily through the User Interface. A key part of this was adding items to basket and ordering. Our automation checks would start by going directly to a product, we would pass in the Product ID as follows:

This was efficient for automation purposes but meant we were skipping one of the most important parts of the customer flow the home page, search and gallery pages, which were the most visited pages of the site.

Our automation also needs to consider user ‘Persona’s’ from a data perspective. Do the users in our automated tests reflect what a real user looks like? Again, in my experience, teams often bulk load in a set of users who are all pretty much exactly the same, again to minimize the overhead of managing lots of different data types. Therefore you’ll often miss issues that occur in the differences between users. A good example is a user returning to a shopping site when they have already added an item to basket. Do all your test start with an empty basket?

To Conclude….

A User Journey is an ’emotional’ experience so we cannot automate these type of tests yet(and probably never will). Ensuring you consider the emotions of your users throughout the entire development life cycle can increase your chances of having an application that will create a good experience for your users. As Testers we can play a valuable part in testing with our users emotions at heart.

Automation can provide an important platform for any customer journey to ensure the application is at least giving the customer the correct result from business logic and allowing to flow through the system.

[1] Introduction to User Journeys

[2] Softbank unveils ‘human-like’ robot Pepper –

[3] Emotions in Software Testing

[4] Shop Direct Test Lab

[5] Testing your Emotions –

[6] Design Thinking –

[7] Testing and Checking refined



To start, a little bit about MobProgramming….

This week(Monday 16th Feb 2015) i attended the XP Programming London Meetup (#xprolo). Benji Webber(@benjiweber) gave a talk about how Unruly have introduced “MobProgramming” into their teams. For those who aren’t familiar with the term Benji defined it as:

“Whole Team focused on same problem using one workstation”

Benji also referenced one of the well known advocates of MobProgramming, Woody Zuill (@WoodyZuill), who’s website has lots of extensive information around the concept if you want to research further –

As the term suggests, “MobProgramming” is primarily focused around the activity of programming. This generally means a group of developers, designing and writing code into a Development IDE, as a group such as in the image below:


On a practical level the team congregate around the screen at an agreed time and start working. The individuals in the group rotate operation of the keyboard around every 10-15 minutes so that everyone contributes code to the solution. The time spent “mobbing” can vary, Uruly have spent whole days doing it but found it can be quite draining working in a whole team activity all day.

The Benefits

Unruly have been doing “MobProgramming” for about 4 months and these are some of the benefits they have observed:

– Its fun!
– More emergent design – Team can design collaboratively as they progress
– Less synchronisation needed as the whole team are present
– Latency is reduced
– Reduction in emails and other offline communication

They have found that some problems don’t support “mobbing” for example things that fall into the Obvious domain in Cynefin. The team discuss and agree when they feel it is sensible to “Mob”. I’d like to think over the time the teams will identify some heuristics on when to “Mob” or use a model as a guide.


Thinking about how “Mobbing” could apply to other activities, it could work well for testing. Lets call this “MobTesting”

The same definition would still be valid:

“Whole Team focused on same problem using one workstation”

Most of the practicalities would also be the same, a big screen, one keyboard, whole team, white board nearby etc. Many of the benefits seen in “MobProgramming” would also be seen.

I do feel there are a couple of subtle differences between “MobTesting” and “MobProgramming”.

Testing is an “explorative” activity.

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.”

We are searching for useful information that we can provide the team and stakeholders to make an informed decision about the quality of a product. “MobTesting” allows us to benefit from having a whole team involved in two ways: Shared Responsibilities and Varied Perspectives.

Actors in the Mob

Firstly looking at Shared responsibilities. In “MobTesting” I see the following actors within the “Mob”:

1. The Driver – The person with the fingers on the keyboard/mouse moving through the application. elderly-driver_2164765b
2. The Note Taker – The person capturing the session. Testing and note taking is incredibly hard to do simultaneously. One of the benefits is that someone can focus on this such as producing a mind map or recording questions. notetaking
3. The Orienteer – The person helping us keep on track and to our charter. Often it’s easy to get side tracked when testing alone so having the benefit of someone to keep the mob focused may help. Antique Compass
4. The Observer – Stands back and observes what’s taking place. Often getting into the detail you can get temporary blindness. Simply observing can help see issues the driver may miss or skip past. Buyer-as-observer-300pxl-96dpi1-215x300
5. The Runner – Synomous with the TV industry. You could use this person to get the refreshments or better, if the Mob has questions for people such as the Product Owner “Is this what you expect?” They can run to the person and get quick feedback. making-tea
6. The Local Expert – Someone who knows the product, domain or technical detail of the Product. This helps if any questions arise from the mob. This person could drop in and out of the session or maybe pulled in at various points by “The Runner” Snip20150212_2

As per Benjis example in “MobProgramning” the individuals doing these roles should rotate every 10-15minutes allowing everyone to keep engaged in the activity. It also provides a good opportunity to practice skills of observation and note taking which can be applied when working alone.

Varied Perspectives

Another great benefit of bringing the whole team together to “Mob” is the varied perspectives on offer, especially if you can include those outside the immediate development team such as customers. If you already have a varied “cross functional” skill set within the Mob this will naturally bring a varied and rich perspective. If not, you can apply critical thinking techniques within the group to ensure a more varied perpsecitve. For example each person in the group could be assigned the ‘persona of a valued person’ such as a customer. You could also apply critical thinking techniques such as Edward De Bono’s “Six Thinking Hats”

“MobTesting” is not a “BugBash”

“MobTesting” shouldn’t be confused with a “BugBash”. I’m sure many testers have been involved in large group bug hunting sessions where the objective is to find bugs. “MobTesting” is different because we are exploring the product as a whole team with the intention of uncovering information, one piece of information could be a bug, but there are plenty more pieces of information that could be found.


If you can do “MobProgramming” and “MobTesting” surely you can just “Mob” anything! I’m not so sure. I think there are certain activities that make sense not to be done in large groups. Things that spring to mind are writing documentation, creating release notes and other similar activities that often fall into the Cynefin “Simple Domain”.

In addition to “MobProgramming” and “MobTesting” another activity that would be interesting is “MobDrawing”. This is where the whole team gather to draw out a design for a new product. In this case the pen could be swapped around the group every 5-10mins.

Watch out for the Mob

It’s important to watch out for bad behaviours when “Mobbing”. These include:

– Bike Shed Effect (’s_law_of_triviality) – Unable to agree on trivial things “such as the colour of the bike shed”

– Dominant Personalities in the group such as “Watch the Master” effect which can often be seen within Pairing

– Group Think – Where people agree for the harmony of the group in order to avoid disagreements

With regards specifically to “MobTesting” you should probably watch out for “Phased Mobbing” For example: Monday = “MobPrototyping”, Tuesday = “MobProgramming”, Wednesday = “MobTesting”

“MobTesting” can be done at any point in the life of a product, in fact you don’t even need a line of code to have been written. You could do “MobTesting” on a paper prototype or a set of wireframes. Testing can also be done in “MobProgramming” and Programming can be done in “MobTesting”. The main difference is that in “MobTesting” we specifically want to focus on a testing problem.

The value of “Whole Team”

Wether we stick “Mob” on the front of an activity or not isn’t really the important thing here. “MobProgramming” and “MobTesting” simply amplify the benefits that can be found when whole teams work together on one problem rather then separately. To exaggerate the point, there is even some evidence that working as a whole team can lead to solutions faster than if people worked apart or in pairs.

You can read more about the Unruly Mob here: