Alois Reitbauer About the Author

Week 23 – 7 Rules to Improve your Application Performance Practices

In this post I discuss the seven most important steps to improve your application performance practices. These simple-to-follow practices will help you to improve the way you deal with application performance. Besides eventually improving the performance of your applications it will help you to avoid playing the classical blame game which normally happens when something goes wrong

Rule 1 – Understand the application

While this may sound trivial and the most obvious thing to do – reality paints a different picture. Most failing application performance management processes originate from people’s lack of understanding of their application. This does not mean that the people involved in the process do not possess the knowledge to solve the problem. The case is rather that they do not have enough data to understand what is really going on in the application itself. They follow an approach which pretty much is like: “I will try to make sense of whatever information I can get.” They see the lack of information as an immutable physical law like gravity instead of trying to get the information they really need. In my post on the Proactivity of Troubleshooting I described in detail which information is needed to understand application problems. You basically must be able to answer the following questions.

  • What has happened?
  • When did it happen?
  • Who is impacted?
  • What is the difference compared to before the problem?
  • Why did the problem happen?

If you cannot answer these questions based on the data you have, then you need to get the data to answer them. Otherwise your activity will be a kind of “performance voodoo” rather than a solid engineering practice. You already think you know where the problem is? Ok, then get the data that proves your guess.

Rule 2 – Measure what Matters

This is the logical consequence of step 1. You have to measure what is important for you and your management. Many people still think that the major goal of performance management is to resolve production problems. This would be equivalent to a CEOs primary job being to save a company that is nearly bankrupt. Most of your work is to avoid problems.

Most people are pretty good in measuring technical aspects of their application like response time of web requests or connection pools. They feel like Captain Kirk (Well, I would prefer Captain Archer) if they have operations dashboards that show a lot of these fancy metrics. However, when they talk to their management they realize that this is not necessarily the information needed by management. In Is there a Business Case for Application Performance discussed this problem in detail.

The concept introduced in this post is Business Transaction Management (BTM). BTM is a specialized discipline within application performance management which targets communicating performance aspects at a business level. Typical questions answered by BTM are:

  • Which user was affected by a production problem?
  • What are the effects of increasing traffic for transaction X by 10 percent?
  • How do we have to change our infrastructure to serve our users better?
  • Why did user “Sam” have a problem accessing his account this morning?

So basically this means that you have to relate your low-level metrics to the context of the application. If you only have measures helping with highly technical problems, you will fail in these higher-level activities important to management.

Rule 3 – Objectify Measurements

As all your activities are based on measurement, you better get them right. The basic rule is that measurement results should be the same irrespective of who is measuring them. In case you think this is a no-brainer, try the following: Ask three different people in your organization how much host CPU is consumed by a certain transaction. Ideally, ask a developer, a tester and an operations guy. (If you really run the experiment I would really be interested in the results – please post them in comments below.)

So the important part here is to objectify the way you measure. This includes the measurement method as well as the tooling. You also have to define how to interpret the measurement. This becomes even more important if you have to work across teams. If you can’t agree on how to measure, how can you ever expect to compare results?

Rule 4 – Define a Language

A prerequisite for talking with each other is talking in the same language. This means you have to establish a language that everybody from management to IT super geeks understand. No, I am not talking about English or whatever your native language is. Have you ever been at the doctor’s and getting an explanation about what’s wrong with you and not being able to figure out whether you are pregnant or growing a third leg? So the import part about a common language is that both parties understand what the other one means.

In management we generally refer to this language as Key Performance Indicator (KPIs). KPIs provide a common means for communication across stakeholders. They do not provide the necessary level of detail you need for your daily work, but they help in coordination and planning. If you now think you have done the job if you define your KPIs as CPU usage, memory consumption and network traffic you did not yet get the point. That’s like a doctor telling all your detailed blood values. Your KPI tells you whether you are healthy or not. You do not care about value x being 20 or 30. In fact you probably have no idea whether a higher value is better or not.

Which values you actually choose depends on your application, but they have to cover things like quality-of-service or provisioning information. BTM is a central concept to collect information at this level. Your more-detailed measures are then used to decide how to influence your KPIs in the direction you need to. If you tell your boss that you cannot serve more than 300 concurrent users and he requests you to serve 1000 you have to figure out how to do that. This will then require to you to look at memory consumption or CPU usage.

Rule 5 – Use a Map and a Compass

So what does this metaphor mean? Can you imagine navigating on the ocean solely with a compass? Already after a short time you will realize that knowing where north is does not really help, if you do not know where you are in the first place. What does this mean for performance management? Performance management has to be a continuous activity; otherwise you get lost and cannot make optimal usage of your measurements. As pointed out in an earlier post (Top 10 Reports are not the final answer) many people think that top 10 reports are the ultimate answer to performance optimization. If I simply optimize the ten slowest parts everything will be fine. While this approach will have the desired effect of performance improvement, it can easily become the wrong way to go. What if what is shown in the top ten report is not the cause of why the application is slower?

So you have to create your map first to know the direction to head towards. In performance management this means to continuously monitor your performance. This enables you to understand trends and whether you are on the right track or have to move in another direction. This goes beyond just measuring the trends in response times. You also have to know why you are moving in the wrong direction. (see rule 1)

When I was doing my open sea sailing certificate, I had to deal with exactly the same issue in navigation. First you need to find out whether you are in the right place or not. If you failed navigating to the right coordinate you have to find out why you got there and how to avoid this in the future by taking into account factors like tides or windward drift.

Rules 6 – Do the simple things first

This is a generally good advice. However, most people do not follow it. As in most cases in life, the 80:20 rule also applies to performance management. You get 80 percent of the success by investing 20 percent of the effort. However it seems to be a law of nature that people are much more attracted by getting the other 20 percent of success. Let me give you some examples.

People try to implement complex high-end caching systems before following simple performance best practices. While all these technologies are great from a performance and scalability point they require massive efforts, while an improved web caching strategy requires nearly zero implementation effort.

Another example is people deciding to start with performance management and trying to get everything fully automated from their CI environment over testing to production. While this is a great goal and perfectly follows the Continuous APM idea this endeavor is doomed to failure. You have to start with regular manual performance analysis first and then step-by-step automate your processes as knowing what to measure is a prerequisite for automation (see rules 2 and 3)

Rule 7 – Every ship needs a captain

A point many people also miss when implementing performance management is to define responsibility. The general rule regarding responsibility is that if it is not clearly defined the result will be chaos. Either many people feel responsible leading to a complete mess or nobody feels responsible leading to … well nothing. The vital step to success is to define who is responsible for performance in your company. This does not mean that he has all the expertise to solve every problem. Most likely he will not and will require the help of other people in the organization. His job is to ensure that all necessary steps will be taken and the right people get involved at the right time.

Conclusion

Performance Management should be part of every company’s software processes. However, failing to follow some important rules will lead to frustration, waste of resources and finally failure. I’ve described the top rules for being successful which solve the major problems I have seen out in the wild. I do not claim this list to be complete, but it goes a long way. If you have important rules to add, feel free to post them as a comment.

If you want to read more articles like this visit dynaTrace 2010 Application Performance Almanac

 

Comments

  1. “Either many people feel responsible leading to a complete mess or nobody feels responsible leading to … well nothing.”

    Having experienced both these scenarios I could not agree more. In fact, often they occur in the same project..nobody considers performance until issues smack them in the face, then all hands to the pumps leading to complete chaos and very little traction in addressing the issues.

    John

  2. Can I add 2 more rules?

    (8) don’t take anyone’s word for it – just because the architect or lead developer says that “it will perform” don’t take their word for it. Test it under load (using tools like Dynatrace to give you that continuous performance view). As we discuss over there, there is some evidence that that’s what Twitter failed to do, hence their current woes… http://blog.siteconfidence.com/

    (9) look out for the “points of inflection” where horizontal and/or vertical scaling stop offering performance improvements and you suddenly have to make major changes. A simple analogy is Excel’s 65K row limit… you are “linearly scaling” up to 65K and then at 65K+1 (the “point of inflection”) it all breaks. Now you can upgrade to Excel 2007 and increase that limit 16x… but in a web world this equates to updating a major version of your database or web server and from personal experience of having updated both across a large web farm this involves lots of planning, application testing, extra “swing” servers etc. This takes quite some time. Time whilst your application performance is suffering and your customers are going elsewhere…

  3. Stephen,

    I have been reading your blog and like the articles. … and yes you are free to add more rules ;-) . I agree with both. Your rule 8 goes well in line with my rules 1,2 and 3. If you have no measurements you are guessing.

    Rule 9 is also a very good one. My experience here is that most people never get to the point of testing at this scale. How many clients do you have that know their “point of inflection”.

  4. Hi Alois,

    Thanks, I get the Dynatrace blog from the PerfPlanet feed and enjoy your stuff too :-)

    Re the “point of inflection” concept I have expanded on this a bit here – http://blog.siteconfidence.com/2010/06/more-on-point-of-inflection.html

    Re our clients, well, I have to say (of course!) that all our load testing clients understand where their application performance curve hits the “point of inflection”. ;-)

    Regarding the original “scaling capacity curve” (where your horizontal/vertical scaling starts to no longer offer an effective ROI and you have to start planning to re-architect everything)… yup, that’s harder to answer.

    I think those customers experiencing rapid growth have a much better eye on this whereas those that have slower growth rates might be less aware of an impending cliff.

    It is worth noting that (from personal experience) that sometimes you KNOW the cliff is there, you know that you are heading towards it, but convincing the management that controls the budget you have to spend millions of pounds to start re-architecting NOW to avoid it can be extremely difficult.

    Justifying the ROI “building a system that is pretty much the same as what we have in terms of functionality but just more scalable” can be a frustrating experience.

    Management would rather spend those millions of pounds on developing new features for the customer, sales and marketing or keeping the venture capitalists happy.

    So the argument is (often) “can we not just throw some more hardware at it” or “can’t you just tweak the application or database to speed it up”?

    The answer is often “yes we can” and maybe we are slowing down the rate we are heading off that cliff… but the cliff is still there… and you may have used up the vital time you needed to build a bridge to your new architecture before you crash and burn!

    -Steve

  5. Great article thanks! I’ve been using OPNET’s ACE Live, ACE Analyst and Panorama solutions for production application monitoring & troubleshooting, but I like your GUI!

  6. This is really a good post.These 7 Rules to Improve our Application Performance Practices are the best i have ever heard and i would try to implement this rules

  7. Really nice post and all of these tips are really helpful to improve a application performance. I will definitely apply all of these rules.

  8. This tips are really nice and they will surely be helpful to improve a application performance, Thanks for sharing this wonderful tips.

  9. The 7 rules are really needed for improving the performance of an application.

  10. This is a very informative article.I was looking for these things and here I found it.

  11. Rule 9 is also a very good one. My experience here is that most people never get to the point of testing at this scale.

  12. The important part here is to objectify the way you measure. This includes the measurement method as well as the tooling. Internet Marketing

  13. This post will surely help me and all person who wants to know about this topic..

  14. A recently conducted study paints a evil picture on Application Performance Practices. 6o percent of the companies admit that they don’t have any performance management processes installed or what they have is ineffective. Half of the companies who answered that they have performance management processes admitted that they are doing it only in a reactive way when issues occur. third of all companies said that management is not supporting performance management properly. Simultaneously companies admitted that performance issues are impacting their business. & ten percent responded that they spend up to half of their development time troubleshooting issues. mensagens de carinho

  15. Management would rather spend those millions of pounds on developing new features for the customer, sales and marketing or keeping the venture capitalists happy.

  16. Keep up the good piece of work, I read few blog posts on this web site and I think that your blog is really interesting and holds lots of great information. technical writing service

  17. Really nice post and all of these tips are really helpful to improve a application performance. I will definitely apply all of these rules.

  18. This is a two part article – also read Combining Analytics with Performance Management Data. I use Google Analytics every day to look at Key Web Performance Metrics to answer questions like:

  19. Article brings up a fascinating point, SEO Nottingham different from what you usually read or what people usually write about.

  20. great and full of various and attractive information about everything. Thanks a lot one more time for the great and informative blog and keep up publishing these great posts in the future. binary options broker

  21. Keep posting the good work. Some really helpful information in there. Great work you have done by sharing them to all. Simply superb. Binary Options Questions

  22. This is the perfect blog for anyone who wants to know about this topic. The article is nice and it is pleasant to read. I have known very important things over here. egypt holidays all inclusive

  23. I use Google Analytics every day to look at Key Web Performance Metrics to answer questions like:sydney funeral services

  24. the import part about a common language is that both parties understand what the other one means.floor painting 0.79 28

  25. cation problems. You basically must be able to answer the following questions.

  26. law of nature that people are much more attracted by getting the other 20 percent of success. Let me give you some examples.Redes de Proteção

  27. Really like your blog content the way you put up the things. I’ve read the topic with great interest and definitely will stick your blog routinely for other great posts. Personalised Childrens Books

  28. jhon devid says:

    This is the ideal blog for anybody who wants to know on this topic. The article is nice and it is enjoyable to read. I have known very significant things over here.
    website development companies

  29. This would be equivalent to a CEOs primary job being to save a company that is nearly bankrupt. Most of your work is to avoid problems. garage Equipment

  30. Really i like your blog component the means business immigration canada you put up the things.I’ve read the topic with great interest and definitely testament stick your blog routinely for other great posts.

  31. I am visit this site first time & it has awesome information. I have come to know a lot of information after visiting this site. Keep this sort of posting in future as fine..

  32. It is really help to us. Its give us lots of interest and pleasure. Its opportunity are dave so so fantastic and working style so speedy. Its really a good article. It gives me lots of pleasure and interest. Dissertation writers

  33. emmaliam says:

    This topic has always fascinated me. Thank you for writing an article that has great content and is well written. Well I am inspired by your writing style.
    free cell phone spy software

  34. Thank you so much for the post you do. I like your post and all you share with us is up to date and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job. work clothes

  35. In the IT world, software applications are being rapidly developed. Clients, and so employers, are just looking for those teams/individuals who can build up applications rapidly, just bothering to make their application live; but what often happens after an application goes live is that users start to use the application and it doesn’t respond well. At this point, clients start to lose users and business. options trading tutorials

  36. Nice blog, thanks for the info. We are one of the UKs leading online suppliers of office products, office supplies, stationery, and office furniture supplies selling an extensive range of over 20,000 products.

  37. adnan saif says:

    Money comes in different forms and denominations. There is paper money and the metallic form which is referred to as coins. The value of money is printed on the surface of the metal or the paper, but in some cases where it is not printed, the worth can be determined in other ways. olive garden coupons

  38. Wow!! What a great writing, really I appreciate such kind of topics. It will be very helpful for us. Waiting for more articles, blogs like this.

Comments

*


− three = 3