Continuous Performance Management

Performance is a very critical non-functional requirement for the SmartClient application that we are building. The application runs on PC's that do not necessarily have the latest spec hardware.  Our application also indirectly "competes" with other company packages that are installed on the same machine. The last thing we want is for our company to loose business due to people complaining that the application is too slow to use.  We therefore generate performance metrics as part of our development process every now and then to ensure that the functionality being added has not degraded the performance of the application down to an unacceptable level.

Process

Our current process of gathering the metrics is however very laborious and also a bit subjective as different individuals measure the timings using a good old fashioned stopwatch approach.  If bottlenecks are identified, we run a profiler against the specific areas to get a detailed level run-down of the code to identify why the application is running slower.  We also profile the application every now and then to try and further optimise certain hotspots.  As most decent profilers allow you to compare snapshots you are able to pinpoint what code is causing the system to slow down.

I've been doing some thinking around this whole performance management (PM) process to see how we can improve the process.  To my mind the ideal PM process should:

1. Require no manual interaction.
2. Generate metrics on the build server as part of our Continuous Integration (CI) Process.
3. Eliminate subjective measuring factors like different operators and doing the measurement using different hardware.
4. Allow performance trends to be spotted easily to allow the team to quickly respond.
5. Cover the complete application.

Improved Process

We've recently expanded our Continuous Integration (CI) Process to automate the Mercury QuickTest Professional (QTP) regression test pack run.  The nice thing about the QTP run is that the application is put through all its paces as the tests are all driven from the UI.  As we already have QTP integrated into our CI process, I'm thinking of using the regression test run to also generate the performance metrics.  The process would involve:

1. Adding performance logging to the application code base.  The log statements will write timing details to a file in XML format.  The logging can be switched on/off using a setting in a configuration file OR one might consider using AOP to inject a performance logging advice OR alternatively have the logs being produced only in DEBUG mode.
2. Use the existing automated QTP runs to create the performance logs. 
3. Write an xsl stylesheet to consolidate the performance logs into a nicely formatted html performance log report.
4. Integrate the performance log as part of the CruiseControl.NET web dashboard for the team to view. 
5. Use the CC.NET Statistics Publisher to generate statistics and some nice looking graphs based on the consolidated metrics.

If we look at my ideal PM process, this seems to cover all of the requirements   The process is completely automated via CC.NET.  The metrics are available for every build.  The metrics are generated without user interaction on the same machine.  Lastly, by using the CC.NET statistics publisher, the team will be able to spot the performance trends quickly and easily.

Feedback

I'd like to know what other people are doing in their environments surrounding performance management?  Is it mostly still a manual process?  Have you tried automating it?  What tools are you using?  What do you feel I should do to further improve the process?  Any thoughts would be much appreciated.