Performance Zone is brought to you in partnership with:

Jeroen Borgers is an independent consultant who has worked with Xebia IT-Architects and Atos Origin in the past. Jeroen helps customers on enterprise Java performance issues and is instructor of Accelerating Java applications courses. He has worked on various Java projects in several industries since 1996, as a developer, architect, team lead, quality officer, mentor, auditor, performance tester, tuner and trainer. He is specialized in Java performance since 2005. Jeroen has posted 3 posts at DZone. You can read more from them at their website. View Full User Profile

Case study: Performance Tuning a Web Shop – Part 2

09.30.2008
| 15582 views |
  • submit to reddit
In the first part of this article, I’ve talked about the speedup challenge of a large Dutch web shop. We discussed our biggest gains applied on bottlenecks found by using evidence. We found the evidence by using tools, most importantly: JMeter for load testing, JAMon for performance monitoring and JARep for performance reporting. In this part we’ll deal with JAMon and JARep and lessons learned.

 

Monitoring with JAMon

When we found a service to be too slow, we wanted to analyze it and identify the bottlenecks. This usually concerned one or more remote call to the database or the mainframe. So we wanted to see which database and mainframe calls were a bottleneck. And since we tested with representative load, a common Java profiler turned out to be not useful most of the time. They cannot deal with JDBC queries or differentiate for parameters specifying the particular mainframe call to make. In addition, they have high overhead and cannot deal with the required load. So, we wanted a different, more lightweight way of measuring here.

Here shines JAMon API, Java Application Monitor. This is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications. See below for a screenshot of our JAMon View page. It shows the chosen counters with statistics like number of hits, average response time, total response time, standard deviation, minimum and maximum response time, which differs from a profiler. A profiler either samples the stacks or is event based. Stack sampling is expensive. An event based profiler instruments chosen methods and is notified when the method is called. It collects and transports this data from the application process to the profiler process or even to a remote profiler process. This data transportation has its overhead, especially when many places are instrumented. JAMon works differently. It does not store all individual samples, just the statistics, in the same VM. So the hits are incremented on method access and the average response time is calculated incrementally. The JAMon results are accessed only on request by e.g. a JSP page. This results in lightweight memory usage and lightweight CPU usage. It actually is most useful in production where we measured an overhead of < 1%.

Figure 1. Screenshot of the JAMon summary page with running statistics for each measured counter.

The newest version of JAMon, version 2.7, comes out of the box with interception of the JDBC driver as well as interception of HTTP requests, for instance in Tomcat. This means that with just configuration you measure all incoming web traffic and outgoing calls to the database. We used the API, which is simple and is basically used as a stopwatch: with start and stop. We make use of Spring AOP to create an Interceptor as seen in the following screenshot.

Figure 2. Invoking JAMon API from a Spring interceptor.

We don’t use the JAMon Interceptor provided with Spring since it is only activated in debug mode, clearly not what we are looking for. The statistics from JAMon now provides us with very valuable information from the actual system behavior in production to isolate bottlenecks.

The next figure shows where we measure with JAMon in the web shop system.

Figure 3. Measuring the Java service layer with JAMon: incoming calls, in between Spring beans and outgoing calls.

 

Reporting with JARep

Since we now had this valuable response time information from production, we also wanted to see how response times change after deploying a new release or a patch. For this, one of us started copy-pasting the counter statistics in Excel and resetting JAMon every morning around 8:00 o’clock. He added some VB script and generated a graph of hits and the average response time values of a set of interesting counters over time. Now we could see the real effects of changes in production, besides other interesting behavior like the weekly pattern and effects of increasing load on response times. However, this approach had some drawbacks like you can imagine. We did not measure in weekends nor holidays, it was error prone, it took Excel longer and longer to process all data and we would like to see with higher resolution than one value per day.

To meet these needs, we developed some pieces of code and a JSP to fetch JAMon data once every five minutes, store it in the database and generate reports from that data. This simple prototype has evolved in what we now call JARep. See the next figure for an overview how JAMon and JARep are deployed.

Figure 4. Deployment of JAMon and JARep for a cluster of four JVM’s.

Management of the web shop IT organization felt that open source tools JMeter and JAMon helped considerable to achieve their performance goals. In return, they offered to donate JARep to the open source community. This also gave them the opportunity to benefit from new development on the tool. I think this is a great example of how open source can and should work and all parties win.

We have put JARep on Sourceforge under the GNU public license. We are currently using it and developing it further for multiple customers of Xebia. We added a Swing client which is deployed with Java Webstart, to get better graphics and a richer user experience in general.

The next figures show screenshots of the jsp page with fictional data based on the actual production data. The user basically chooses the counters to show, time period to show, the time resolution, the diagrams to show, toggle aggregation and toggle ignore peaks. The first result screens show the four counters called ‘verwerkenOrder’ (processOrder.) These are the fully classified names prefixed with JVM instance alias. By aggregating, the average is taken of those four instances for average response time, the maximum of the four is taken for the maximum time, and the sum of the four is taken for the total time and number of hits. This gives us the complete picture for the cluster.

If we want to see the values for each JVM individually, we just don’t aggregate, as seen in the following screenshot.

Another useful feature is to show the top counters, like the top 10 for average response time, or the top 5 of total time. See next figure. It shows the top 10 of total response time from January until March. It can clearly be seen that the introduction of newer, faster hardware for the database on the 5th of March has its effects on response time, e.g., the green line.

We are currently working on a number of things like notifications when thresholds are exceeded. An item on the wish list is support for not just elapsed time, but more generic, any value. This is already supported by the newer versions of JAMon. For instance order amount to see (near) real time sales.

 

Key lessons learned

The key lessons learned are as follows.

  • For predicting response times in production and preventing problems, it is important to simulate user behavior in a representative way. You need to avoid unreal caching effects. You can achieve this by load testing with Apache JMeter.
  • For seeing real application behavior in production, be able to isolate bottlenecks and be able to react to incidents quickly, it is important to monitor actual response times of application components in production. This can be achieved by using JAMon.
  • For discovering changes over time, like with the introduction of a new release or long term trends, it is important to store performance statistics on component level over time and be able to analyze them visually. This further facilitates finding bottlenecks, quick resolution of incidents and enables to be prepared for the future. This can be achieved by use of JARep.

 

Conclusion

Open source tools Apache JMeter, JAMon and JARep each greatly facilitated our performance tuning approach for the web shop. Thanks to these tools we could tune based on evidence and speedup crucial parts of the application with a factor of ten. Moreover, the customer IT organization now has a process in place using these tools to systematically assure performance by testing, tuning and monitoring their web shop.

 

About Jeroen Borgers

Jeroen Borgers is a senior consultant with Xebia - IT Architects. Xebia is an international IT consultancy and project organization specialized in enterprise Java and agile development. Jeroen helps customers on enterprise Java performance issues and leads Xebia's performance practice. He is instructor of Java Performance Tuning courses. He has worked on various Java projects in several industries since 1996, as a developer, architect, team lead, quality officer, mentor, auditor, performance tester, tuner and troubleshooter. Jeroen is specialized in Java performance since 2005. He is an internationally acknowledged speaker on that topic and he regularly publishes articles. You can read more on the Xebia's blog at http://blog.xebia.com.

 

Published at DZone with permission of its author, Jeroen Borgers.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Pieter van der Meer replied on Wed, 2008/10/01 - 2:45am

Jeroen,
although i think the solution is workable and a stable one (provides good insights). I'm suprised you did not go to standard tooling like JMX. For monitoring applications with JMX there are quite a few products available. commercial and open source. And a respecting company has tools like HP OpenView or Tivoli. Building an application based on propriatry tool is a maintenance nightmare for the future. I personally think that standards like JMX are here to stay and provide a much more stable solution. combine this with for example the Annotation features that Spring provide (coming up in JMX 2.0), and its much more powerful and easier to maintain. Pieter

Jeroen Borgers replied on Wed, 2008/10/01 - 4:25am in response to: Pieter van der Meer

Hi Pieter, 

Thanks for your remarks. We have considered exposing JAMon data by JMX. Difficulties with this are that several customers still work with rather old appservers like WebSphere 5.1 with java 1.4.2. These environments have a rather limited JMX support. Furthermore, we did not see a suitable way to expose the JAMon data by JMX in order for it to be understood and well presented by a JMX based monitor tool. However, this was some time ago, things might have changed over time and I am open to suggestions. In addition, many companies do not have such a well working monitoring tool which could present the jamon data in a useful way. And if they have such a tool, I doubt it will be easy to configure to get the value out that JARep provides out of the box. 

I don't really understand what you mean by 'building an application based on a propriatery tool is a maintenance nightmare'. I find that to be posting FUD. Like explained in the article, we used Spring AOP to apply JAMon measuring as cross cutting concern. The application depends on the JAMon api from only one or a few interceptors, which can be taken out by configuration. So, the functionality of the application does not at all depend on the jamon api nor on the jarep tool. And further, how do you measure 'maintenance nightmare'? I have not seen any need for maintenance in applications because of development/changes in jamon or jarep. And in my experience that cannot always be said for those great 'stable' Java standards we have, like for instance EJB. And JMX is evolving, too.

That said, I think JMX is a good way to integrate with other monitor tooling and if there is a need or if it adds value for the customer, I find it a good idea to seriously consider leveraging jmx in jamon and/or jarep. And since these are open source, we can all have our influence here. 

Most importantly, in my experience, the customer just wants a tool that works and helps them in firefighting and preventing performance problems of their applications. And that is was jamon and jarep have proven to provide.

 --Jeroen.

Pieter van der Meer replied on Thu, 2008/10/02 - 3:07am in response to: Jeroen Borgers

Jeroen,
I was imposing a FUD, and i did not read the article in all the details. I missed the part about the Spring AOP to insert the JAMon as a crosscut. With that i think the solution is clean and stable for the future. The main point i wanted to make is why not JMX.
The maintenance nightmare i'm talking about is caused by similar measuring projects a was involved with. And choices where made to use "tiny" open source projects that are no longer supported. When you say that the application is developed some time ago, the maintenance is taken care of.
Pieter

David Gillespie replied on Wed, 2008/10/15 - 10:32am

Jeroen,

 

You mention in your article that you are using Jamon to monitor applications in a cluster.  I have not read any information suggesting Jamon has this capability currently; how do you acheive this, and can you get a wholistic view of the operations across the cluster, or just on a node by node basis?

 

David.

Jeroen Borgers replied on Wed, 2008/10/15 - 1:26pm

Hi David,

 

We monitor each JVM in the cluster individually with JAMon. With JARep we collect this data as shown in Figure 4 and are able to see one or several JAMon counters of all JVM's together in one graph. This is shown in the second screen shot with graph. There is also the option to aggregate over all JVM's, as also shown in the first screen shot with the graph with only the red line of the 'processOrder' counter. 

Jeroen. 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.