Andreas Grabner About the Author

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

How to use your Selenium Tests for automated JavaScript/AJAX Performance Analysis

With more than 3 million downloads, Selenium is the world’s most popular functional testing framework. It allows web developers and QA professionals to automatically test how an application functions on multiple browsers, such as Firefox, Internet Explorer and Chrome and across operating systems, such as different versions of Windows and Mac OS.

The FREE dynaTrace AJAX Edition on the other side is probably the best web site performance analysis tool for Internet Explorer 6, 7 and 8.  Here is what John Resig, Creator of jQuery has to say about it: “I’m very impressed with dynaTrace AJAX Edition’s ability to get at the underlying “magic” that happens inside a browser: page rendering, DOM method execution, browser events, and page layout calculation. Much of this information is completely hidden from developers and I’ve never seen it so easily collected into a single tool. Huge kudos to dynaTrace for revealing this information and especially so for making it happen in Internet Explorer

Combine best of breed tools for Automated Performance Analysis

If we look at these two tools it almost seems logical to use them in combination in order to automatically capture performance data from within the browser for every Selenium Test you execute.

Why would you want to do that?

In case your Selenium Tests identify a problem – whether it is a functional issue (page not working correctly) or a performance problem (page are much slower in the latest build) – the data that is automatically captured by dynaTrace AJAX contains all information necessary for your web engineers to figure out the root cause of the problem. Without this information, your engineers probably go ahead and manually reproduce the error by executing and debugging through the test case manually. Manual work means additional time and effort – and that’s what can be avoided. Best of all – BOTH TOOLS ARE FREE!!

How it works – Example on testing Google Search

Selenium has multiple options to execute tests. Either use your HTML Test Suites that define the individual test cases in HTML Tables or use a programming language such as Java and “code” your tests. Let’s have a look at these two scenarios and how you can get Selenium to work with the dynaTrace AJAX Edition.

The integration basics are explained in Automation with dynaTrace AJAX Edition and  in Advanced Timing and Argument Capturing. The dynaTrace AJAX Agent that runs as an Internet Explorer Add-On gets activated and is configurable via several environment variables. DT_IE_AGENT_ACTIVE=true activates the Add-On so that it collects data from the current browser session. DT_IE_SESSION_NAME can be used to specify the recorded session name, e.g.: MySeleniumTest123. DT_IE_SERVER_HOST and DT_IE_SERVER_PORT can be used to connect to a dynaTrace AJAX Edition client that runs on a different machine – this is useful when you want to centralize data capturing. DT_IE_CLEAR_CACHE=true causes the Add-On to clear the browser cache before any actions are executed – this is very useful when you want to test your web sites cache behaviour.

These environment variables need to be set for the browser instance that runs the tested website. The browser is launched by the selenium server process. Therefore we have to set these environment variables for that server process as they are propagated to all processes that get launched from it.

Requirement: The dynaTrace AJAX Edition must be running on the local machine in order for the launched browser instances to connect to it. You can also run the AJAX Edition on a different machine – in this case you need to specify DT_IE_SERVER_NAME with the name of the machine where the AJAX Edition runs on.

Running an HTML Test Suite

The following batch file sets the necessary environment variables to a) activate the dynaTrace AJAX Add-On in IE and b) specifiy a custom session name:

@echo off
set SELENIUM_RC=C:\selenium\selenium-server.jar
set SCRIPT_PATH=%~dp0

rem dynaTrace AJAX IE agent settings
set DT_IE_AGENT_ACTIVE=true
set DT_IE_SESSION_NAME=Google_SeleniumTest

java -jar %SELENIUM_RC% -log exec.log -htmlSuite "*iexplore" "http://www.google.com"
  "%SCRIPT_PATH%\GoogleSuite.html" "%SCRIPT_PATH%\GoogleResults.html"

Before I run this script I make sure to launch the dynaTrace AJAX Edition. When I execute the script Selenium will launch the HtmlRunner in a separate browser instance and will then launch the browser that will execute your test steps. The second browser instance will have the active dynaTrace AJAX agent. You can tell this by the changed IE icon as well as an increasing number in the Captured Events displayed in the dynaTrace AJAX Edition IE Toolbar:

Executing an HTML Test Suite with dynaTrace AJAX capturing performance data

Executing an HTML Test Suite with dynaTrace AJAX capturing performance data

Note: Even though both browser instances use the DT_IE_XXX environment variables, the AJAX Edition will not collect information from the Selenium Test Runner. The Test Runner runs as an HTA Application which prevents the AJAX Edition Add-On from being loaded. In this case it is an advantage as we are not interested in the performance of the Selenium Runner.

Once the test is done and the browser is closed we can explore the results captured in the dynaTrace AJAX Edition:

dynaTrace AJAX Edition captured performance data from the Selenium Test Run using the configured Session Name

dynaTrace AJAX Edition captured performance data from the Selenium Test Run using the configured Session Name

The session name is the one we passed in via DT_IE_SESSION_NAME – Google_SeleniumTest in this case. You will see a URL to a local file. That is a file that Selenium opens which then actually opens the URL of the test application. Simply ignore this line and focus on those URLs of your test scenario.

Running Test Case from Java (or JUnit)

Many folks out there use the option to write their Selenium tests in Java – either using JUnit as executing framework or just using the DefaultSelenium proxy in a console app or custom test framework. The approach here is the same as explained in the Html Test Case Use Case. We have to start the selenium server with the DT_IE_XXX environment variables. Here is the batch file I use:

@echo off
set SELENIUM_RC=C:\selenium\selenium-server.jar
set SCRIPT_PATH=%~dp0

rem IE agent settings
set DT_IE_AGENT_ACTIVE=true
set DT_IE_SESSION_NAME=dynaTrace_Selenium

java -jar %SELENIUM_RC%

Here is my Java application that executes the same Google Search test steps as my Html Test Suite:

public static void main(String args[]) {
  Selenium selenium = new DefaultSelenium("localhost", 4444, "*iexplore", "http://www.google.com");
 
  selenium.start();
  selenium.open("http://www.google.com");
  selenium.type("q", "dynatrace");
  selenium.click("btnG");
 
  selenium.stop();
}

The result is the same. The Selenium server launches two browser instances. The first loads the Selenium Runner which won’t be analyzed by dynaTrace AJAX as it is an HTA Application. The second launches google.com and searches for dynaTrace. The following screenshot shows this captured session with the name dynaTrace_Selenium:

dynaTrace AJAX Edition captured the data from the test steps executed by my Java application

dynaTrace AJAX Edition captured the data from the test steps executed by my Java application

Where to go from here?

The feedback I get from our user community is that this type of integration is a huge time saver. Whenever their Selenium tests fail, the in-depth performance data from the test run is available for analysis. A key point to remember is that this is not just about performance. As the AJAX Edition captures full JavaScript traces, DOM Access, Network and Rendering activities, it makes it very easy to identify all functional and deployment-related problems. Analyzing the dynaTrace AJAX data allows one to identify whether the problem is caused by files that didn’t get deployed on the web server, whether a new JavaScript library was introduced that wasn’t fully tested yet, or whether the problem actually originates on the server-side code. Check out the following additional resources to learn more about how to analyze the data that is captured: Getting Started with dynaTrace AJAX Edition (includes links to real-life web site analysis examples), Why Web 2.0 requires full End-to-End visibility and Learn how Monster.com uses the AJAX Edition for their web-site

In case you have questions on this topic, either comment on the blog or use the comment feature on our dynaTrace AJAX Community Portal.

 

Comments

  1. that’s all cool & great post to read, thank you very much for sharing.

  2. I like Selenium very much, it is a great tool with only one problem – poor browser support. It works reliably with IE and FF only.

    For cross browser compatibility I use it along with BrowserSeal – http://browserseal.com

  3. AJAX is not a new programming language, but a new way to use existing standards.

    AJAX is the art of exchanging data with a server, and update parts of a web page – without reloading the whole page.Roll forming machine

  4. What about Selenium Grid and dynaTrace? Is it possible?

  5. @Tiger: you can run dynaTrace on any machine where you execute Selenium Tests. With the environment variables you can control whether dynaTrace should be active or not.

  6. When I run Internet Explorer manually the IE icon is changed and dynaTrace agent works as it was intended (both variables DT_IE_AGENT_ACTIVE, DT_IE_SESSION_NAME are defined) and session is correctly traced. But when I run IE using Selenium RC and Python Script dynaTrace agent doesn’t work in a proper way, and there is no session log in dynaTrace.

  7. Hmm. Now it works.
    After selenium server restart.

  8. you have to start selenium with those environment variables as selenium is going to launch IE and is then passing these variables to the newly launched IE processes. an alternative would be to set these variables globally on the machine which would however result in every IE instance to become active.
    makes sense?

  9. Vlad M. says:

    Hi, Andreas!

    From the article you’ve published I understand this way of measuring the performance as: watching for an application with a stopwatch in the right hand – the difference is that it’s a script watching for an app with a very powerful stopwatch :)

    I think such an approach appear quite useful for determining big problems with performance that appear even without significant load on the app. But with this method it appears I can hardly look into problems that appears under high load only.

    It would be very interesting for me if I could do the same measurements for the high-loaded application.
    Of course I can put a back-ground load on application using some performance test tools (LoadRunner, SilkTest, JMeter, etc) and on top of that measure performance of an AJAX app with Selenium+dynaTrace, but it still will be a bit inaccurate measurement because some AJAX calls make the load itself which is not emulated in background.

    In regard to this, might be you have some idea how to launch your solution in many parallel threads, so I can put a load of, say 500, virtual users?
    Sorry if question is too stupid, I’m quite unfamiliar with both Selenium and dynaTrace :) .

    Thank you!

  10. Hi Vlad – there are no stupid questions – just stupid people that don’t ask :-)

    To answer your questions
    a)look into SilkPerformer’s new Browser Driven Testing – http://blog.dynatrace.com/2010/05/04/elevating-web-and-load-testing-with-microfocus-silkperformer-diagnostics-powered-by-dynatrace/ – with that feature you can run a load test that is actually driving a browser instance for each Virtual User
    b)you can use a service such as BrowserMob that runs Selenium tests as load-tests. We could talk with Patrick Lightbody (the creator of Selenium and BrowserMob) if it would be possible to add dynaTrace AJAX to that MIX
    c)you could go with the approach you described – run some HTTP load generators and then run some Selenium Scripts to test AJAX Performance. I am sure you are also aware of the feature that allows you to run Selenium in your environment on multiple machines so that you can actually generate some load with Selenium as well

    hope that helps. stay tuned with our blog posts – we always come up with some crazy ideas to solve problems like yours :-)

  11. Vlad M. says:

    Thanks Andreas for the detailed reply!

  12. What about Selenium Grid and dynaTrace? Is it possible?

  13. Alex Chen says:

    Hi Andreas Grabner:
    I use Selenium to test,but not have network report,I don’t know why?can you help me ,thanks.

  14. @Alex.Please use the dynaTrace AJAX Community Forum (https://community.dynatrace.com/community/display/AJAX/dynaTrace+AJAX+Edition+Community+Home) for discussions like this – also – check the troubleshooting page: https://community.dynatrace.com/community/x/rgcSAQ

  15. Hi Andreas Grabner,
    This article is very helpful.. couple questions.
    1. After the session information is captured in the dynatrace, How do I save this session info from automation. Either from selenium or dynatrace programmatically.
    2. How do I Insert markers while running the tests in dynatrace?

    Thanks in Advance.

  16. @Saritha
    The sessions are automatically captured by dynaTrace. So – after the test is completed you have all information available under the Sessions Node. dynaTrace also provides a commercial offering for Web Performance Analysis where we give you a full REST interface to query data, compare, baseline and trend web performance data.
    As for the markers. Check out the following blog: http://blog.dynatrace.com/2010/04/13/advanced-timing-and-argument-capturing-with-dynatrace-ajax-edtion/

  17. Hi Andreas,
    the link “Automation with dynaTrace AJAX Edition” seems to be broken.

    Keep up the good work!

    Greetings to Alois, I think we’ve met in person a few years ago at SAP :-)

    Someone should extend Selenium with a Standard API (similiar to BrowserMob) for performance tests …

    Regards,
    Markus

  18. Greeting to you too Markus :-) Yes, we met at JavaOne talking about the Post Mortem JSR.

    Regarding the Standard API: Let’s see what happens next after the Webdriver API Integration.

  19. @Markus: thanks for letting me know about the broken link. its fixed

  20. I went to download selenium and there are a number of versions available. Should I be using the RC version since Dynatrace Ajax only runs on IE?

    Regards

  21. @Peter: RC allows you to run your scripts against Internet Explorer and allows you to activate dynaTrace AJAX. Check out our other blogs on this topic that Alois and I recently published. Let us know if there are any further questions

  22. Thank you very much for the in depth guide, this will help immensely when I go to set up my own server and network.

  23. Greetings to Alois, I think we’ve met in person a few years ago at SAP

    Someone should extend roll forming Selenium with a Standard API (similiar to BrowserMob) for performance tests …

    Regards,
    Markus

  24. Of course I can put a back-ground load on application using some performance test tools (LoadRunner, SilkTest, JMeter, etc) and on top of that measure performance of LED Lightbar an AJAX app with Selenium+dynaTrace, but it still will be a bit inaccurate measurement because some AJAX calls make the load itself which is not emulated in background.

  25. Hi,
    Great work guys..Awesome plugin for my selenium tests.

    I am using WebDriver and my Java code looks like,
    WebDriver driver = new InternetExplorerDriver();
    driver.get(url);
    ..//call for login test
    Adding following lines in my batch file is not activating the dynatrace plugin when my selenium tests are launched.

    set DT_IE_AGENT_ACTIVE=true
    set DT_IE_SESSION_NAME=dynaTrace_Automation

    I also added following two calls but still no luck :( what am I doing wrong? Right now, I am manually activating the plugin. Would appreciate any help.

    set DT_IE_SERVER_HOST=localhost
    set DT_IE_SERVER_PORT=9988

  26. Hi NDK
    Can you verify that these environment variables actually get passed to IE? You can use Process Explorer for this. What I also found is that you should launch WebDriver tests as Administrator. Can you try executing your batch file using that option?
    For future questions I also recommend using the dynaTrace Forums – http://forums.dynatrace.com – to address questions like this.
    Cheers

  27. Thanks Andreas. I will check those out. If I still have issues will post the question in the forum.

    Again awesome work.

    Cheers

    • Hi Ndk, do you post the question in the dynaTrace forum? I was also confused with this, If you did it , could you show me the link of the issues in forum? Thanks.

  28. food flavor says:

    Of course I can put a back-ground load on application using some performance test tools (LoadRunner, SilkTest, JMeter, etc) and on top of that measure performance of an AJAX app with Selenium+dynaTrace, but it still will be a bit inaccurate measurement because some AJAX calls make the load itself which is not emulated in background.

  29. How do I Insert markers while running the tests in dynatrace?

  30. You could go with the approach you described – run some HTTP load generators and then run some Selenium Scripts to test AJAX Performance. I am sure you are also aware of the feature that allows you to run Selenium in your environment on multiple machines so that you can actually generate some load with Selenium as well
    hope that helps. stay tuned with our blog posts – we always come up with some crazy ideas to solve problems like yours

  31. I went to download selenium and there are a number of versions available. Should I be using the RC version since Dynatrace Ajax only runs on IE?
    Regards

  32. dynaTrace AJAX now also runs on Firefox

  33. I went to download selenium and there are a number of versions available. Should I be using the RC version since Dynatrace Ajax only runs on IE?
    Regards

  34. I went to download selenium and there are a number of versions available. Should I be using the RC version since Dynatrace Ajax only runs on IE?

    Regards

  35. It doesnt matter which version of Selenium you use. dynaTrace will work with any functional testing tool that drives a browser – whether it is IE or FF. Firefox support has been added a while ago

  36. Hi,
    Could you please help me.

    1) I can run automation selenium tests in Internet Explorer with dynaTrace.(The dynaTrace AJAX Agent that runs as an Internet Explorer Add-On gets activated and is configurable via several environment variables.
    @echo off
    set SELENIUM_RC=C:\selenium\selenium-server.jar
    set SCRIPT_PATH=%~dp0

    rem dynaTrace AJAX IE agent settings
    set DT_IE_AGENT_ACTIVE=true
    set DT_IE_SESSION_NAME=Google_SeleniumTest)

    2) But I can not run it in FireFox. No sessions were opened in DinaTrace for FF. How should we do it? May be we should use another environment variables DT_FF_AGENT_ACTIVE? Can we do it in dynaTrace-AJAX-edition Version: 3.2? May be we should use only PREMIUM version of dynaTrace in this case?

    I used this blog: “How to use your Selenium Tests for automated JavaScript/AJAX Performance Analysis”

    http://blog.dynatrace.com/2010/05/21/how-to-use-your-selenium-tests-for-automated-javascriptajax-performance-analysis/

    Thanks.
    Yuriy

    PS I used
    - dynaTrace-AJAX-edition Version: 3.2.0.900 , built on 2011-10-11
    - IE 7.0
    - FF 3.5.7
    - Selenium 2.0(selenium-server-standalone-2.11.0.jar )

  37. I went to download selenium and there are a number of versions available. Should I be using the RC version since Dynatrace Ajax only runs on IE?
    Regards

  38. Greeting to you! This article is very helpful.. couple questions.
    1. After the session information is captured in the dynatrace, How do I save this session info from automation. Either from selenium or dynatrace programmatically.
    2. How do I Insert markers while running the tests in dynatrace?
    Thanks in Advance.

  39. @Tonychen: dynaTrace AJAX Edition also runs on FF

    Every session is stored automatically for you. For further questions and similar topics I suggest checking out our discussion forum on this. go to http://apmcommunity.compuware.com – logon with your dynaTrace Community Account and check out the disucssion forums

  40. How to automate an hta application using selenium webdriver or any seleniumm driver? Please help.

  41. Connor Gilbert says:

    Hey when I run the script on my machine Selenium automatically opens the 64bit version of IE by default which is not compatible with our plug ins.

    When I put the location of the exe in directly then the selenium scripting window never appears.

    Any idea how to fix this?

    Thanks

Trackbacks

  1. [...] the original post: How to use your Selenium Tests for automated JavaScript/AJAX … No [...]

  2. [...] How to use your Selenium Tests for automated JavaScript/AJAX … [...]

  3. [...] How to use your Selenium Tests for automated JavaScript/AJAX … [...]

  4. [...] How to use your Selenium Tests for automated JavaScript/AJAX … [...]

  5. [...] How to use your Selenium Tests for automated JavaScript/AJAX … [...]

  6. [...] just posted a really nice tutorial showing how to track client-side performance (JavaScript, DOM, render times, etc) automatically [...]

  7. [...] H&#959w t&#959 &#965&#1109&#1077 &#1091&#959&#965r Selenium Tests f&#959r automated JavaScript/AJAX … [...]

  8. [...] URLs. The browser instances are started with the dynaTrace agent enabled. Read how to do this here. After the test has finished the results are automatically uploaded to the configured ShowSlow [...]

  9. [...] already blogged about how to automatically start the dynaTrace agent from Watir or Selenium. In fact it is nothing more than setting two environment [...]

  10. [...] How to use Selenium Tests for programmed JavaScript/AJAX Performance Analysis [...]

Comments

*


− 3 = three