Thursday 5 March 2009

Performance Testing Approaches

Nearly all performance testing is based on protocol simulation. This can be a neat, manageable, efficient way to performance test. Notably most tools in this market have the capability to performance test HTTP based applications. Some tools make a dogs dinner of script creation (usually open source), some are elegant & effective (usually commercial), and some hide everything under the covers including errors & mistakes etc and take a script-less approach (also usually commercial and in my view always best avoided).
HTTP lends itself very well to protocol level testing and most enterprise applications will use HTTP in some form or another. Of course not all applications use HTTP, there are still many that use other protocols such as ODBC, SQLNet, RMI and more. There are tools that can capture these protocols I can think of at least three, however in my experience the parameterisation of these scripts can be a daunting task for even the most seasoned automation expert. It can be time consuming, require expensive specialists and can fail. Also if the application is re-written, then the script often has to be too, causing further expense.
It leads me to ask the following question:
Could it be better to have the machines deal with the complexities of protocol chit chat and performance test from the GUI?

Here’s my view and I hope this will spurn further discussion.
Pro’s: Saves time & effort, eradicates errors in scripts, makes regression testing for performance less of a head ache, reduces the need for specialists, independent of protocol therefore can be a one fits all approach.
Con’s: Needs a lot of hardware (terminal servers and / or virtualisation), each piece of hardware / virtual hardware needs a license so can be costly, can be clunky in execution, synchronisation issues become more prevalent.

Are we in a situation now where hardware is so cheap and available that we should be looking at farms of servers to help us with our performance testing, rather than relying on a neat but expensive script based approach?

No comments:

Post a Comment