Klaus Enzenhofer About the Author

Klaus is a Senior Strategist in the Center of Excellence at CompuwareAPM. Klaus influences the strategic direction and development of application performance management solutions.. He has deep experience gleaned from years of developing and running large scale web and mobile applications for online businesses.

Comparing User Experiences: Web, App & Mobile

With the establishment of mobile web and native mobile apps, your website isn’t the only customer touch point anymore. Mike Austin, Director of e-Commerce at RoomsToGo said it best:

Our customers expect the same experience throughout all our channels no matter if it is the website, mobile site or the real physical store!”

So, how can you measure the user experience across the various channels your customers use to interact with you? Let’s see what we can do to make the user experience quantifiable so we can make informed data-driven decisions.

If we look at the different channel types, they have all two things in common – a user and actions that the user executes. To make this more visual I put the graphic below together. It depicts a common Visit of a user with what their common User Actions might look like across each of the channels (Mobile App, Mobile Web & Desktop Web). Although the screens look different the expectations are always the same.

Visit Journey with User Actions for Mobile App Mobile Web and Desktop Web: All visitors have the same expectations when booking a vacation, regardless of the channel they choose to use

Visit Journey with User Actions for Mobile App, Mobile Web and Desktop Web: All visitors have the same expectations when booking a vacation, regardless of the channel they choose to use

Why you Should Look at User Actions?

User actions are the important element as it exists on the web as well as on the mobile app. It is therefore something we can compare between the different channels. In the above graphic the second user action is “Search for… ” The user expects that the time it takes from the touch/click on the search button till the result list is displayed is similar. This timing is now our first indicator for the User Experience; the User Action Duration. We can now compare the user action duration over all the different channels and can work on getting the experience to the industry best practice “3 second promise”.

In addition to the User Action Duration, errors have a huge impact on the user experience. No matter if it is a client side error like a Crash or JavaScript Error, an error occurring in your datacenter like a “Null Pointer Exception” or a Third Party Service not being available, the impact on the user and your business is huge. Some developers try to disguise the errors by presenting “funny” error screens but the negative impact is still there.

Screenshot of error message in a iOS App that is user facing

Screenshot of error message in a iOS App that is user facing

 

The 404 Error Message delivered from Livadaru.net might be funny to look at, but the potential impact on your users is nothing to laugh about! See more funny examples at http://www.1stwebdesigner.com/inspiration/unique-404-error-pages-inspiration/

The 404 Error Message delivered from Livadaru.net might be funny to look at, but the potential impact on your users is nothing to laugh about! See more funny examples at http://www.1stwebdesigner.com/inspiration/unique-404-error-pages-inspiration/

Looking at the user action duration and the number of errors, we now have two good metrics to quantify the user experience for each user action. But we still have more to investigate to get the overall user experience.

What Additional Information do we get by Looking at a Visit?

The collection of experiences across all user actions is what ultimately defines whether the user experience was satisfied, tolerated or frustrated. If we look at a “visit” as a collection of all user actions in the order the user executed them,  we can get four very important data points: the first action, the last action, did the visit bounce and if the visit converted. These all help us to identify the User Experience Index. The screenshot below shows us an example of a single visit with every User Action completed. The user started the app, searched for a vacation, tried to book the trip and then the app crashed.

Details of a single visit– showing all actions the user did with a mobile app crash at the end

Details of a single visit– showing all actions the user did with a mobile app crash at the end

I have seen numerous customer examples like the one above where visitors had perfect experiences until they hit the final steps. Typically, if the last action fails or takes too long for the user, it will cause the user to bounce off to your competition. Therefore we have to consider a user that had bad performance or was facing an error on the last action, as frustrated.

Alternatively, I recommend you pay special attention to the first action of the visit. You have only one chance for a first impression and if your app starts slow or your landing pages contain JavaScript errors, the user will leave very quickly. As the visit contains all action including your conversion step, we can now also tie the conversion back to a landing page and see which once work best. In addition we can look specifically at bouncing visits with just a single action.

Landing pages dashboard – showing which landing pages exist, which ones are popular, which ones produce high failure rates and bounce rates

Example: Which landing pages are popular? Which ones produce high failure rates and bounce rates?

These metrics are also very valuable when it comes to A/B or multivariate testing. Sometimes changing a picture or color does not help because errors and bad performance are already destroying the user experience. If you want to know more about it I wrote a blog focusing on why you’re A/B testing efforts might not have any chance to succeed.

 

How to Collect the Data

The collection mechanisms for the different technologies might vary, but there are Real User Monitoring solutions  available that allow you to collect the real user monitoring data from:

  • Mobile Native Apps
  • Mobile Hybrid Apps
  • Desktop Web Browsers
  • Mobile Web Browsers

Do what you can to present the data gathered in a way that presents the most value to you. Feel free to use the image below as an example, when attempting to correlate the metrics we talked about here to the overall user experience. If you want to try it out on your own then checkout our dynaTrace Free Trial Edition.

Channel Key Metrics Comparison dashboard: Comparing number of visits, User Experience Index, Conversion Rate and Bounce Rate over the three different channels Desktop Web, Mobile App and Mobile Web

Channel Key Metrics Comparison dashboard: Comparing number of visits, User Experience Index, Conversion Rate and Bounce Rate over the three different channels Desktop Web, Mobile App and Mobile Web

Continental User Experience Overview dashboard – Showing from where the users are connecting, what user experience they have and overtime changes in load time and traffic per continent

Continental User Experience Overview dashboard – Showing from where the users are connecting, what user experience they have and overtime changes in load time and traffic per continent

Conclusion

Delivering the same user experience across all the channels (Mobile App/Mobile Web/Web) is critical to the success of your business. By looking at the satisfaction of a visit, you’ll get a better understanding of what the user’s real experience was. Don’t just look at the performance and faults that occur on a single action but also at the critical entry- and exit actions. Considering these metrics, you can easily compare the overall experience of your users using one or multiple channels.


Comments

  1. Klaus,

    I read your article in its entirety and felt like I am your clone sitting here in the Bay Area thinking and working along the same lines. I have a few more suggestions for you to add on top of your ideas.

    Please contact me at vijay.prathap3@gmail.com

    Vijay

Comments

*


six + 2 =