Andreas Grabner About the Author

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Tips on creating stable Functional Web Tests to compare across Test Runs and Browsers

In the last week my task was to create stable functional tests for a new eCommerce application. We picked several use cases, e.g.: clicking through the different links, logging in, searching for products and actually buying a product. We needed functional tests that run on both Internet Explorer and Firefox. With these tests we want to make sure to automatically find any functional problems but also performance and architectural problems (e.g: too many JavaScript files on the site, too many exceptions on the server or too many database statements executed for a certain test scenario). We also want to find problems that happen on certain browsers – therefore the task of letting these tests run on the two major browsers.

Test Framework: Selenium WebDriver

As test framework I decided to pick Selenium WebDriver and downloaded the latest version. I really thought it is easier to write tests that work in a similar way on both browsers. I have several lessons learned

  1. When you write a script always test it immediately on both browsers
  2. Use a Page Object approach when developing your scripts. With that you keep the actual implementation separated from the test cases (you will see my test scripts later in this blog – will make more sense when you see it)
  3. Be aware of different behaviors of IE and FF
  4. Make sure your test code can deal with unexpected timings or error situations

What a test script should look like (slick and easy to understand)

Here is a screenshot of one of my test cases.

Selenium Test using Page Object Pattern that allows writing easy to understand test cases

Selenium Test using Page Object Pattern that allows writing easy to understand test cases

 

Common functionality in PageObjectBase

I put lots of helper methods in a base class that I called PageObjectBase. As WebDriver doesn’t provide functionality to wait for certain objects or for the page to be loaded (at least I haven’t found anything on that) I created my own waitFor methods to wait until certain objects are on the page. This allows me to verify whether my app made it to the next stage or not. Here is another screenshot of one of my helper methods. You see that I had to work around a certain limitation I came across in IE – seems like By.linkText doesn’t work – same is true for most of the other lookup methods in By. What worked well for me is By.xpath with the only limitation that certain methods such as contains() don’t work on Firefox. As you can see – lots of things to consider – not everything works the same way on every browser :-(

Helper methods in my PageObjectBase class

Helper methods in my PageObjectBase class

 

Easy to switch Browsers

My test classes create the WebDriver runner. Here I also created a base class that – depending on a system property that I can set from my Ant script – instantiates the correct WebDriver Implementation (IE or FF). This base class also checks whether dynaTrace will be used to collect performance data. If that’s the case it creates a dynaTrace object that I can use to pass Test and Test Step Names to dynaTrace. This makes it easier to analyze performance data later on – more on this later in this blog.

Base Object that makes sure we have the correct WebDriver and that dynaTrace is properly set-up

Base Object that makes sure we have the correct WebDriver and that dynaTrace is properly set-up

 

Analyzing Tests across Test Runs

As recently blogged, dynaTrace offers Premium Extensions to our free dynaTrace Ajax Edition. These extensions allow us not only to collect performance data automatically from Internet Explorer or Firefox – it automatically analyzes certain key metrics per test case. Metrics can be the number of resources downloaded, the time spent in JavaScript, the number of Redirects or the number of Database Queries executed on the Application Server.

Identify Client Side Regressions across Builds

I have access to different builds. Against every build I run my Selenium Tests and then verify the Selenium Results (Succeeded, Failed, Errors) and the numbers I get from dynaTrace (#Roundtrips, Time in JS, #Database Statements, #Exceptions, …). With one particular build I still got all successful Selenium Test executions but got a notification from dynaTrace that some values were outside of the expected value range. The following screenshot shows some of these metrics that triggered an alert:

JavaScript errors, number of resources and number of server-side exceptions show a big spike starting with a certain build

JavaScript errors, number of resources and number of server-side exceptions show a big spike starting with a certain build

A double click on one of the metrics of the build that has this changed behavior opens a Comparison View of this particular test case. It compares it with the previous test run where the numbers were ok:

The Timeline makes it easy to spot the difference visually. Seems we have many more network downloads and JavaScript executions

The Timeline makes it easy to spot the difference visually. Seems we have many more network downloads and JavaScript executions

A Side-by-Side Comparison of the Network Requests is also automatically opened showing me the differences in downloaded network resources. It seems that a developer added a new version of jQuery including a long list of jQuery plugins.

When these libraries are really required we need to at least consider consolidating the jQuery library and using a minified version of these plugins

When these libraries are really required we need to at least consider consolidating the jQuery library and using a minified version of these plugins

Now we know why we have so many more resources on the page. The best practice recommends that we merge all CSS and JavaScript files into a single file and deploy a minified version of it instead of deploying all these files individually. The JavaScript errors that were thrown were caused by incompatibility between the multiple versions of jQuery. So – even though the Selenium Test was still successful we have several problems with this build that we can now start to address.

Identify Server-Side Regressions across Builds

Even though more and more logic gets executed in the browser we still need to look at the application executed on the application server. The following screenshot shows another test case that shows a dramatic growth in database statements (from 1 to more than 9000). Looks like another regression.

Database executions exploded from 1 to more than 9000 for this particular test case

Database executions exploded from 1 to more than 9000 for this particular test case

The Drill Down to compare the results of the problematic build with the previous works in the same way. Double click the measure and we get to a comparison dashboard. This time we are interested in the database statements. Seems it is one statement that got called several thousand times.

This database statement was called several thousend times more often than in the previous test runs

This database statement was called several thousand times more often than in the previous test runs

When we want to know who executed these statements and why they weren’t executed in the build before, we can open the PurePath Comparison Dashlet. The PurePath represents the actual Transactional Trace that dynaTrace captured for every request of every Test Run. As we want to focus on this particular database statement we can drill from here to this comparison view and see where i’ts been called.

Comparing the same transaction in both builds. A code change caused the call of allJourneys as compared to getJourneyById

Comparing the same transaction in both builds. A code change caused the call of allJourneys as compared to getJourneyById

 

Analyzing Tests across Browsers

In the same way as comparing results across test runs or builds it is possible to compare tests against different browsers. It is interesting to see how apps are different in different browsers. But – it is also interesting to identify regressions on individual browsers and compare these results with the browser that doesn’t show the regressions. The following screenshot shows the comparison of browser metrics taken from the same test executed against Internet Explorer and Firefox. Seems that for IE we have 4 more resources that get downloaded:

Not only compare metrics across test runs but also compare metrics across browsers

From here we can go on in the same way as I showed above. Drill into the detailed comparison, e.g.: Timeline, Network, JavaScript or Server-Side execution and analyze the different behavior.

Want more?

Whether you use Selenium, WebDriver, QTP, Silk, dynaTrace, YSlow, PageSpeed or ShowSlow – I imagine you are interested in Testing and you want to automate things. Check out my recent blogs such as those on Testing Web 2.0 Applications, Why you cant compare execution times across browsers or dynaTrace Ajax Premium.

Comments

  1. Good post.

    Why do you throw an exception instead of a junit Assert on error ?

    and the com.thoughtworks.selenium.Wait class can be used to replace the old clickAndWait functions from selenium 1

    JavaDoc here:
    http://selenium.googlecode.com/svn/trunk/docs/api/java/com/thoughtworks/selenium/Wait.html

  2. Hi Dave – great input on the Wait Helper class – didnt know about it.

  3. Well explained Andreas, also thanks for Dave for Wait helper class.

  4. Great to see that it works for others, at least. :-(

    I threw out all Selenium tests because they were unusable. I had ~80% false positives; tests would fail because Selenium claimed it could not click a link that was clearly there. Run the same test 5 times and once it passes and 4 times it fails, telling me it can’t find the link it’s supposed to click.

    I tried everything: sleeping and pauses, trying 5 times, upgrading Seleniumm…

    My guess is that Icefaces and Ajax are at least part of the problem, but the bottom line remains:

    Selenium is a waste of time.

Comments

*


four − 2 =