Andreas Grabner About the Author

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Top Performance Mistakes when moving from Test to Production: Deployment Mistakes

In my first blog I covered Excessive Logging as being a top performance problem when moving an application from test to production. In this blog I talk about common deployment mistakes that lead to functional as well as performance problems. Whether it is missing files that lead to HTTP 4xx, misconfigured Web Server settings that severely impact performance and throughput or JavaScript problems that happen on browsers that weren’t tested. These are all common mistakes that can easily be avoided. Let’s have a closer look.

Example #1: Missing Files in Deployment impacts User Experience

When deploying an application into production it is important to not forget to deploy all content of that application. That includes all static resources such as CSS, JS or Image files. The following screenshot shows the Errors users are experiencing because of several missing JavaScript files. The data.js file is used to allow users selecting a date from a calendar control instead of entering the full date manually. In this case – because the JavaScript file was not correctly deployed – end users that click on the calendar icon will not get to use that feature:

Analyzing real user traffic reveals missing JavaScript files that were not correctly deployed

Analyzing real user traffic reveals missing JavaScript files that were not correctly deployed

Missing JavaScript files also lead to JavaScript Execution errors and may impact more than just one feature that is now no longer available to the end user. The following shows the JavaScript errors that happen in the browser for real end users when there are problems with missing or problematic JavaScript files:

JavaScript Errors captured from real users reveal the actual impact of JavaScript file deployment issues

JavaScript Errors captured from real users reveal the actual impact of JavaScript file deployment issues

How to prevent this before moving to Production

Problems like this can’t always be found in a traditional load testing environment for two reasons:

  1. Load tests are often done on HTTP Protocol level only and therefore not always download and execute all content that a real browser will download
  2. Load tests that are executed in house don’t test whether content is correctly deployed via CDN’s

It is therefore recommended to

  1. Deploy your application to your production environment (or a staging that is accessible from the web). With that you make sure that you test in a realistic environment
  2. Execute a full set of tests against the application to verify deployment works correctly
  3. As a lot of static content will typically be served by a CDN (Content Delivery Network) execute load from different locations around the world. With that you make sure that ALL content is downloadable from ALL important geographical regions.
  4. Some Cloud-based testing services additional offer executing their scripts using real browser replay. With that you make sure that you test your page as requested by real end users and not just by the generated load testing script.

Example #2: Incorrect Web Server Access Settings impacts User Experience

Files that can’t be accessed because of incorrect access restrictions lead to similar problems as explained in the previous example. When moving from Test to Production many of our customers run into the problem of already existing access restrictions for certain URL patterns, e.g: it is not allowed to download JavaScript files from the root directory or there might be specific security restrictions on certain URL patterns in place. Unfortunately these restrictions are often not well communicated. Therefore an application that worked well in the test or staging environment will either break or experience an additional performance impact for most end users because of these existing restrictions.

The following screenshot shows a situation where an application that is supposed to run without any security restrictions got deployed on a server with existing restrictions in place for the URL pattern /js and /images producing > 14.000 HTTP 401 Requests within minutes that did not occur in the test environment:

An existing access restriction on certain paths caused 14000 HTTP 401s within several minutes after deploying to production

An existing access restriction on certain paths caused 14000 HTTP 401s within several minutes after deploying to production

If we have a closer look and analyze the full Page Action we get to see an even more interesting picture. The user requests the initial Form.aspx page without any authentication restriction problems. The browser then tries downloading the individual JavaScript and CSS files from the restricted locations. All of them coming back with an HTTP 401 Unauthorized Response. The browser accepts the authentication challenge and resends the request for each of these resources – this time including the user security context. This leads to an extra roundtrip for every resource that is in the restricted folders and with that impacts page load time and therefore User Experience:

The Page Action shows all requests necessary to download the page. The restricted content causes an additional roundtrip and impacts performance

The Page Action shows all requests necessary to download the page. The restricted content causes an additional roundtrip and impacts performance

How to prevent this before moving to Production

Similar to the previous example it is recommended to test this deployment on the production system instead of just testing it on a test or staging environment. This makes sure you become aware of any restrictions that exist. You would install the application on your production hardware configuring, configuring your load balancers so that no real user gets access to the new version until the real Go-Live. The recommended steps therefore are

  • Deploy the application on a set of your production environment with proper load balancing configuration so that only your tests can access these servers
  • Run Load Tests – preferable using real browser replay. This will test how real browsers deal with downloading these resources which makes it easy to detect deployment issues related to access restrictions
  • Watch out for any missing JavaScript, CSS or Image files that might be blocked by Firewalls or Web Servers due to existing restrictions

Example #3: Slow or Misconfigured Web Server Modules/Extensions

Web Servers (Apache, IIS, …) provide a long list of Modules and Extensions that handle tasks such as RedirectUrls, Compression, Authentication, Filtering, …

It is important to only use modules that are really needed. It often happens that modules and extensions are installed and activated for every application that is served by that web server. Where one module might be required by Application A it might not be needed by Application B. Therefore make sure to only activate those modules for a specific application that are really required. When adding a new module/extension also make sure it is designed to handle the load that you expect. Common problem that we’ve seen is that extensions are downloaded from the web which were created for small demo projects where they worked well but which have never been tested in large scale production environments.

Identify problems like this by watching high times between Web Server Tier and Application Server. The following Transaction Flow shows that 90% of the transaction time is spent between IIS handling the request natively before passing it on to the ASP.NET Engine:

A slow Module in IIS responsible for URL Rewrites severely impacts request response time

A slow Module in IIS responsible for URL Rewrites severely impacts request response time

How to prevent this before moving to Production

It comes back to the same as said in the previous examples. Make sure to test on your production environment with proper load. Additional check the following things:

  • Review the Web Server Modules/Extensions that are currently configured. If your application runs alongside other applications on the same web server make sure that your application doesn’t inherit settings from other applications
  • Reach out to the vendors/authors of modules you plan to use and verify that these modules have been tested in high load environments
  • Execute your Tests on the production system – only with that you make sure that you test your application as it will run once accessible to the end user. Analyze the time spent on the Web Server as shown in the screenshot above

Additional Reading

If you are interested in Load Testing also check out my post on To Load Test or Not to Load Test: That is not the Question.

Comments

  1. How did you generate the Transaction Flow diagram above? manually? or is there a tool for that?

  2. Normand Poirier says:

    That is Dynatrace’s blog, the graphics you see were generated using their software.

Comments

*


eight − 2 =