Andreas Grabner About the Author

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Do more with Functional Testing – Take the Next Evolutionary Step

Functional Testing has always been an activity done by Test Engineers using Functional Testing tools like QTP from HP/Mercury, SilkTest from Borland/Segue, Functional Tester from IBM/Rational, …
But time is changing – so is Functional Testing – and the things you can do with it. In this article I talk about

The Past: Traditional Functional Testing

All those tools traditionally had their record/replay approach by recording the interactions with the User Interface that should be tested as well as recording the control structure of the windows/controls that were tested. With this separation it was easier to maintain scripts in case the layout of the application under test changed. You only had to update the description of the UI and not each test script that used that description.

The drawback of all these tools was that it required a special set of skills to do Functional Testing. Each tool came with its more or less proprietary testing language as well as IDE and runtime. These were some of the reasons developers usually didn’t get in-touch with functional testing. Due to the lack of exposure developers also frequently questioned the results of functional testing tools as they were presented to them by the Testing Engineers leading to built up tension between the testing and development teams.

The Present: The Functional Testing world is changing to …

Over the past couple of years – greatly driven by the Open Source Community as I believe – we could see a move towards bringing Functional Testing closer to the developers. The approach taken was to provide testing frameworks that allow writing functional tests in Java, .NET, JavaScript, … just as you would write your unit tests using frameworks like JUnit or NUnit.  The frameworks are easy to use and provide support for the major browsers, RIA frameworks and desktop UI technologies.

The list of available frameworks is growing - a lot of them are free, e.g.: Selenium, WebAii, WindMill

Benefits of Functional Testing Frameworks

I see multiple benefits of these frameworks:

  • Developers can now easily write functional tests besides only writing unit tests
  • Functional tests can cover test areas where it is hard to write unit tests for
  • Functional tests can easily be integrated in your Continuous Integration process -> execute them as you execute your unit tests
  • Developers not only think about how components can be made testable for unit testing – but also how the UI can be made testable for Functional Testing
  • BETTER TEST COVERAGE
  • Reuse tests for load-testing and production monitoring

The Future: Go beyond Functional Testing with Transactional Tracing

Functional Testing in the hands of developers promises to bring us more tests that can easily be integrated in the existing automation processes like Continuous Integration. So – is there anything missing? YES THERE IS

  • Root Cause Analysis: you still have to sit down and figure out why certain tests failed. This often requires manual re-runs of the tests with an attached debugger to track down the problem. Too often the problems cannot be reproduced on the developers machine but only on the test machine that was setup with a special environment – which might just not be available right now
  • Architecture Validation: functional testing as well as unit testing only verifies the functional correctness of the tested components. It does not verify if the underlying code is efficient and scalable when executed under load or in production like environments

The next evolutionary step for Functional Testing is to combine Functional Testing with Transactional Tracing. Analyzing the executed transactions from end-to-end down to the invoked components,  across tiers and runtime boundaries allows you to

Speed up Root Cause Analysis

  • you get the dynamic execution path of the tested code as part of the functional test result for each functional test step
  • no need to manually re-run tests as all code-level execution results have already been collected on the target machine where it happened
  • no need to debug through code as the transactional trace contains method level information including execution time, arguments, exceptions, …

Automate Architecture Validation

  • you can analyze the dynamic code behaviour in terms of e.g.: number of sql-queries, number of remoting roundtrips, memory allocations, …
  • you can analyze which components really interact with each other and if the dynamic execution matches what is on the blueprint
  • Automating the analysis of these “Architectural Rules” allows you to let test runs fail in case certain rules have been violated even though the application might be functionally correct
  • identify problems early on that would later cause problems under load or in production

How it works with dynaTrace and WebAii

ArtOfTest offers an easy to use automation framework for .NET. As part of their upcoming 2.0 version (which is currently in beta) it will be possible to register a callback for every request/response that is sent by the browser while executing the test script. This callback enables us to add the dynaTrace HTTP header to the HTTP request allowing dynaTrace to link the original test request with the resulting PurePath. Furthermore we can log the unique PurePath ID that was captured to the unit test output as the HTTP Response header contains this unique ID. This ID can then be used to open the PurePath in question in case a problem occurred.

I will write a more detailed blog entry about how the callback has been implemented and how you can use it. Here are some screenshots that show how to enable a WebAii testing script with Transactional Tracing using dynaTrace and how it can be used to improve Root Cause Analysis as well as to automate Architecture Validation.

Step 1: Writing a test script and enabling the dynaTrace Integration

WebAii test in C#. Enabling Integration with dynaTrace

WebAii testcase in C#. Enabling Integration with dynaTrace in Setup step

Step 2: After running the test – analyze the recorded transactions in dynaTrace

dynaTrace shows transactions with the names used in the test steps

dynaTrace shows transactions with the names used in the test steps

Step 3: Speed up Root Cause Analysis by by looking at the PurePath (=Transactional Trace) that captured the problematic test case

Analyze failed test steps by drilling into the execution details

Analyze failed test steps by drilling into the execution details

Step 4: Perform Architectural Analysis by looking at the PurePath data showing service-to-service interactions, roundtrips to the database, … – all for individual transactions. Automate that process by defining Alerts on Architectural Rules

Architectural Analysis of individual test steps

Architectural Analysis of individual test steps

Conclusion

We have come far with Functional Testing but there is still more that we can do with it. Adding Transactional Tracing into the mix allows you to speed up problem resolution as well as finding problems early on through Functional Testing.

Comments

  1. Hi there

    Have a look at Canoo WebTest, a test automation tool for functional tests. WebTest provides great reporting functionalities and allows you to drill down to the last error on each page.

    Regards,
    Tomi

  2. Canoo is a great tool. We have actually integrated it with dynaTrace and use it for Web Transaction Monitoring. This means that we can define Canoo WebTest Scripts that are then executed on a scheduled basis against your production environment. We capture the results from Canoo, e.g.: Response Times, … and capture the transactional results that we get from within the Application. This integration gives us a complete transaction trace from Canoo all the way to the backend system.

    Cheers
    Andi

  3. I agree with your premise entirely, and I’d like to add an additional benefit for having programmers develop their own functional tests – It helps elevate our focus from the minutiae to the purpose of the software that we are writing.

    Programming is the act of transforming requirements into something that a computer can perform – and the more that programmers keep their focus on the functional requirements, the more likely their implementation of the requirements will be.

Trackbacks

  1. [...] Do more with Functional Testing – Take the Next Evolutionary Step … [...]

  2. [...] please check out the blog postings about Performance Management in Continuous Integration and Do more with Functional Testing. Also check out our White Papers about Continuous Performance Management. [...]

Comments

*


five + = 14