Andreas Grabner About the Author

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

How to do Security Testing with Business Transactions – Guest Blog by Lucy Monahan from Novell

Lucy Monahan is a Principal Performance QA Engineer at Novell, and helps to manage their distributed Agile process.

One of the most important features of an application is to provide adequate security and protect secrets held within. Business Transactions used with Continuous Integration, Unit-, Feature- and Negative Testing specialized for security can detect known security vulnerabilities before your product goes to market.

Catch Secrets Written to Logging

Plaintext secrets written to a log file are a well-known vulnerability. Once an intruder gains access to a hard disk they can easily comb through log files to further exploit the system. Here a Business Transaction is used to search logging output to look for secrets.

Application data slips into logging in a variety of ways: lack of communication regarding which data is a secret, lingering early code, perhaps before a comprehensive logging strategy has been implemented, old debug statements or perhaps inadvertent inclusion via localization processing.

It’s a good idea to grep log files after your test run but that will not cover output to the server console, which may contain different content. For example, users may start the application using nohup which may write terminal logging to nohup.out. And starting an application with redirection of stdout and stderr will persist console output to a file, such as:

startserver.sh > /opt/logs/mylogfile.log 2>&1

Use Business Transactions to search logging content during your testing and trap secrets before they are revealed! And be sure to enable a high logging level since trace level may contain secrets that info level does not.

This Business Transaction uses two Measures for two of the org.apache.log4j.spi.Log4J LoggingEvent class constructors. Other methods that can be trapped include those from proprietary logging classes or classes associated with the auditing channel or retrieval of localized messages.

These Measures simply search the constructor’s message argument for the word “testuser” for purposes of demonstration:

Create the two Argument Measures for the two different Logging Constructors

Create the two Argument Measures for the two different LoggingEvent Constructors

The Business Transaction Filter defines both Measures with an OR operator to activate the Business Transaction:

The Business Transaction will filter transaction that contain a log message for testuser

The Business Transaction will filter transactions that contain a log message for testuser

We want to know about any instance of a secret being logged thus the Threshold Upper Severe is set to 1.0 occurrence. Each of the Business Transactions outlined here use this Threshold.

The threshold on the Business Transaction can later be used for automatic alerting

The threshold on the Business Transaction can later be used for automatic alerting

When the secret is written to logging the Business Transaction is activated:

The Business Transaction Dashlet shows us how many log messages have been written that includ our secret text

The Business Transaction Dashlet shows us how many log messages have been written that include our secret text

The displayed columns in the Business Transaction can be customized to show counts of each Measure that matched the filter.

TIP: Running this Business Transaction against Install and Uninstall programs is highly recommended since a lot of secret data is requested and used during these processes.

From the Business Transaction Dashlet we can Drill-Down to the actual Transactional Trace (PurePath) and see where this secret was actually logged. The PurePath contains additional contextual information such as HTTP Parameters, method arguments, exceptions, log messages, database statements, remoting calls, …

The actual transaction that contains our captured secret. The dynaTrace PurePath contains lots of contextual information valuable for developers

The actual transaction that contains our captured secret. The dynaTrace PurePath contains lots of contextual information valuable for developers.

 

Catch Exception Messages Containing Secrets

If your application prints Exceptions and their messages then you need to catch any secrets embedded within them. This Business Transaction uses a Measure to obtain the return value for java.lang.Throwable.getLocalizedMessage(). Depending on your application’s architecture a Measure for java.lang.Throwable.getMessage() may also be required.

With the Business Transaction defined, perform a negative testing test run that intentionally throws exceptions.

The Measure in this example simply searches for the word “authentication” in the getLocalizedMessage() return value for demonstration purposes:

Argument Measure that counts the occurences of "authentication" in the return value of getLocalizedMessage

Argument Measure that counts the occurrences of “authentication” in the return value of getLocalizedMessage

Business Transaction that filters based on the captured secred in getLocalizedMessage

Business Transaction that filters based on the captured secret in getLocalizedMessage

Business Transaction Dashlet showing us instances where "authentication" was part of the localized message

Business Transaction Dashlet showing us instances where “authentication” was part of the localized message

Catch non-SSL Protocols

Any non-SSL traffic over the network has the potential for exposing secrets. Most applications requiring high security will disallow all non-SSL traffic. Your application needs to be tested to ensure that all requests are being made over SSL. Non-SSL connections can inadvertently be opened and used when different components within an application are handling their own connections or oversight in installation or configuration allowed the connections.

Unless you include an explicit check for non-SSL connections then they may be opened and used stealthily. One way to ensure that only SSL connections are being used is to trap any non-SSL connections being opened.

This Business Transaction uses a Measure to obtain the String value of the first argument to the constructor for javax.naming.ldap.InitialLdapContext. The first argument is a Hashtable and when SSL is enabled one of the entries contains the text “protocol=ssl”.

It’s worth noting that the presence of the “protocol=ssl” value is particular to the JVM implementation being used. Indeed, when SSL is not enabled the protocol value is omitted, hence the use of the “notcontains” operator for this Measure. Use a Method Sensor to capture the value being used in your JVM implementation to confirm the text token for your application environment.

Measure that evaluates whether SSL is not passed as argument

Measure that evaluates whether SSL is not passed as argument

Business Transaction that filters those transactions that do not pass SSL as protocol argument

Business Transaction that filters those transactions that do not pass SSL as protocol argument

Business Transaction Dashlet shows the transactions that do not use SSL as parameter

Business Transaction Dashlet shows the transactions that do not use SSL as parameter

What other protocols are being used in your application? You can write a similar Business Transaction to ensure that only SSL is being used for that protocol.

Catch Secrets Written to Print Statements

The above examples catch secrets written to Exceptions and logging. But a common debug technique is to use a print statement, such as System.out.println() in Java, rather than a logger function. A Business Transaction that catches print statements is recommended for the test suite.

Having said that, in Java java.lang.System.out.println() is not accessible in the usual way because “out” is a field in the System class. An alternative approach may be to use a Business Transaction with a Measure based on “Classname value” to trap all calls to java.lang.System. All related PurePaths for this Business Transaction would then merit review to assess security risks. This approach may or may not be feasible depending how often your application calls java.lang.System. Feasibility testing of this approach would thus be necessary.

The test application used here does not call java.lang.System so an example is not included. In any case, a companion test that performs a full text search of the source code tree for calls to java.lang.System.out.println() is highly recommended.

Summary

For each Business Transaction the process is:

  • Define which entities within your application represent secrets (e.g. passwords, account numbers)
  • Define which classes and methods will be used for the Measure definition
  • Create a new Business Transaction using the Measure definition
  • Ensure that the input data used in the test contains many secrets

Include these types of Business Transactions in continuous integration, unit, feature and negative testing that contain a diversity of secrets. These Business Transactions are not intended for load testing, however, since load testing may simply contain many instances of the same secret rather than a diversity of secrets and the result will flood your result set.

Security-related issues released into the field are costly. The customer’s sensitive data is at risk and the expense of security field patches and the damage done to the corporation’s reputation is high. Understanding your application’s architecture will help identify possible vulnerabilities and enable you to grow a suite of Business Transactions designed specifically to trap secrets and guarantee your application’s security.

Follow Up Links for dynaTrace Users

When you are a dynaTrace User you can check out the following links on our Community Portal that explain more on how dynaTrace does Business Transaction Management, how to improve your Continuous Integration Process and how to integrate dynaTrace in your Testing Environment.

Comments

  1. Joel Scambray says:

    If you are really interested in finding security issues with web apps, you shouldn’t be forcing that feature out of a performance product. That is silly.

    Detecting the issues discussed here and the dozens that were not mentioned (XSS, SQLi, blind SQLi, remote file include, local file include, CSRF, Response splitting, session hijacking, session fixation, resource eumeration, etc) is not trivial and there exists an entire class of programs which analyze source code, object code, or executing programs specifically for these programs. Better yet, specialized tools will do a much better job reducing false positives (very common in code analysis tools) and false negatives.

    Take a look at tools like Fortify, Veracode, Ounce Labs, or Armorize.

    While this article shows you *can* (sort of) find security issues with dynatrace, it would be disingenuous to present this as a realistic practice that QA should spend it time using.

  2. Michael Demastrie says:

    You are right, Joel, this is not a substitute for a comprehensive security testing plan and the focused tools that support it, but I think you’re missing Lucy’s point. Maybe the title is a bit of an oversell. Dynatrace is not just a performance tool; it is a powerful tool with many uses beyond profiling. It’s perfectly good for this kind of targeted testing, especially if you know the kind of parsing you want to do. I always tell people that Dynatrace is dangerous without a skilled operator and such a skilled operator would understand the code under test and the limits of this testing.

    This use case is a creative application of Dynatrace. It highlights very useful features like protocol filtering, exception parsing, and print output tracing. Many people probably don’t realize these features exist. I don’t think Lucy or Andreas ever meant to imply Dynatrace was a Nessus replacement. But, to be honest, many of the security tools don’t do a great job at identifying password or username leaks in log output in live systems, so this usage might very well have a good place in supplementing them.

    What really concerns me is your last sentence. It seems to indicate someone needs to make sure quality engineers aren’t experimenting or using their tools creatively. That would be a dark world to live in. Push the limits, I say, and defy those who say not to. But, to inject your point, understand what you’ve done and what you didn’t do.

Comments

*


2 + five =