Low cost performance testing

Performance testing has typically been an expensive business with the pricing standard set by Mercury (now HP) with their LoadRunner product. This has seen a proliferation over the last few years of tools vendors of this type in the market.
Recently we launched a new type of license - a pay as you go license which makes load and performance testing available at a very digestible cost.
It's been very interesting for consultancy companies to enable them to provide performance testing as a service under their own brand.
For now we're not going to publish information about this on the web site as many ideas in this market are simply copied, if you'd like to know more then please contact us at ukinfo@quotium.com, or come and see us at the free UKCMG show in London, or SQS, also in London (see our events pages).

Performance Testing Approaches

Nearly all performance testing is based on protocol simulation. This can be a neat, manageable, efficient way to performance test. Notably most tools in this market have the capability to performance test HTTP based applications. Some tools make a dogs dinner of script creation (usually open source), some are elegant & effective (usually commercial), and some hide everything under the covers including errors & mistakes etc and take a script-less approach (also usually commercial and in my view always best avoided).
HTTP lends itself very well to protocol level testing and most enterprise applications will use HTTP in some form or another. Of course not all applications use HTTP, there are still many that use other protocols such as ODBC, SQLNet, RMI and more. There are tools that can capture these protocols I can think of at least three, however in my experience the parameterisation of these scripts can be a daunting task for even the most seasoned automation expert. It can be time consuming, require expensive specialists and can fail. Also if the application is re-written, then the script often has to be too, causing further expense.
It leads me to ask the following question:
Could it be better to have the machines deal with the complexities of protocol chit chat and performance test from the GUI?

Here’s my view and I hope this will spurn further discussion.
Pro’s: Saves time & effort, eradicates errors in scripts, makes regression testing for performance less of a head ache, reduces the need for specialists, independent of protocol therefore can be a one fits all approach.
Con’s: Needs a lot of hardware (terminal servers and / or virtualisation), each piece of hardware / virtual hardware needs a license so can be costly, can be clunky in execution, synchronisation issues become more prevalent.

Are we in a situation now where hardware is so cheap and available that we should be looking at farms of servers to help us with our performance testing, rather than relying on a neat but expensive script based approach?

How to define performance testing requirements

It really depends on the kind of business the application is supposed to fulfil the requirements of.
In my experience most of the time there are two types of organisations:
- The type that roughly know how their users are going to behave
- The type who have no idea what to expect from their users
It's really easy to start to calculate the requirements for the type of organisation that has a captive set of users. Take for example a call centre for an insurance firm. They know they have to be able to deal with X sales enquiries every day. They also know that Y clerks will be connected to the application on a typical or busy day. You then know the concurrent user rate and the transaction rate. Therefore the application must be able to support this business scenario. X users and a Y transaction rate.
You have to do this kind of evaluation for every type of scenario that is deemed high risk or high volume, typically there will be 5-30 scenarios.
For the organisation that knows nothing about their users such as an organisation creating a new web site / application and opening it to a new market then you have to start with what you can do. That might be to find out how many transactions the application can support per hour under a given load, thereby finding the limits of the application – you’ll be surprised how few web apps do actually scale as I’m sure people who use proper tools have found out. *Note: Beware the tools market (open source & cheap) – I’ve seen some real horror stories using these tools relating to false positives. I’m definitely not saying use LoadRunner (in fact I’d actively avoid it) but do look around and run some evaluations.

It happens to the best of us!

Today this story made it to the general press (Metro) - even google make mistakes.

Application performance testing: Modeling load testing scripts

by Alberto Saynes - originally posted on blog.quotiumtechnoloies.com 23rd July 2008
Automated Load testing is used to seek the answers to many different types of questions at different stages in the development and deployment of an application. Before a first deployment in production, these questions might be:
“What load will my application support for a given level of performance?”
“How will my application perform in time under nominal load?”
“How will my application react to spikes in load levels?”
“What changes do I need to make to reach a given level of performance at higher load levels?”
“What do I need to do to increase the performance of my application?”
“How does the new version perform as compared to the last one?”“How would the application react to loss of IT resources?”…
Once the application is in production, the questions are more pressing and the time to resolve performance issues is critical. The main questions at this stage are
“Why is the application not delivering the expected performances?”
“What needs to be done to quickly remedy performance issues in production?”
These two sets of questions are of course very much related. However, the application performance management tools currently available on the market do not make it easy to leverage the pre-production test runs and results when in production or to feedback production performance data to better design further load testing runs. A first level of response to this issue can be offered by using the same performance monitoring tool both in pre and post deployment in order to quickly identify and debug performance issues. A more comprehensive way to address this issue is to use integrated products to link the load tests executed pre-production with the application performance monitoring set in place in production. Very few companies offer this type of integrated solution. Whatever the load testing and performance monitoring tools, one key element allowing to leverage any load test in production is the simulation of real-life loads, a result of successful script Modeling.
All load testing tools provide a more or less elaborate script recording function. Recording scripts however is only the starting point. In order to simulate useful real life loads, each script must be modeled to include user-related profiles, variables, and actions. How modeling is supported by the script editor included in a load testing tool is a critically important factor in productivity and in the overall value of your load testing activity.
Modeling includes all script modification and customization that will lead to different user behavior based on which virtual user executes that script. It also makes it possible for a virtual user to react appropriately to dynamic application responses. Modeling techniques might include algorithms developed to reflect statistical behavior observed in production, Variables introduced to make user actions dependant on dynamic application responses or other techniques. However, for most comprehensive web applications, some modeling is actually necessary for the script to function at all. One of the trickiest and most time consuming scripts modeling activity involves dealing with Session Identifiers.
Session identifiers are used by the application to uniquely identify a client browser, while background (server-side) processes are used to associate the session identifier with a level of access. Thus, once a client has successfully been authenticated to the web application, the session identifiers can be used as a stored authentication voucher so that the client does not have to retype their login information with each page request. Manually dealing with session IDs can be a lengthy script edition task (even with powerful parsing and editing functions), and it is prone to errors.
Models: High end Load testing tools supports the concept of model in their script editor. Models are sets of script edition rules rules which can be created by the user and re-used anytime to automatically edit any scripts with a simple click. For the most common session ID modeling requirements, the editor should come loaded with pre-configured, ready to use models. These should include one click automated modeling support for application such as:
.NET applications (__VIEWSTATE, __EVENTARGUMENT,… variables),J2EE applications (jsessionid)SAP applications (sap-wd-cltwndid, sap-wd-appwndid variables)Siebel applications (SWEC, _sn, SWETS,… variables)Epiphany applications (SGloginRequest,… variables).
Etc…
Using these readily available models considerably speeds up the script modeling process, reduces the numbers of editing errors or omissions, and saves precious time to focus on application specific modeling so that the simulated load reflects real-life use of the application.

Snow causes web site & service failure?

Before I go on I'd like to make it clear that neither I nor the company I work for are in anyway related to National Rial or their web site and that they are simply, being used an example.

During the recent snow in London this monthI became one of probably many furstrated mobile web users.

Most mornings I use my phone to check the satus of the train service running from my local station. Orange provide a pretty good service for this - in fact they feed this service directly from the National Rail web site, however as transport on that fateful February (2nd of) morning was seriously diminished, it seems everyone had the same idea and subsequently saturated the service with requests for information.

I then tried to access the national rail enquiries web site and found of course that that too was unavailable. In fact normal service did not resume until later that afternoon. I found this to remain the case for the following few days every morning - an obvious effect of the internet activity of the commuting public.

So what can be done about this?

Effective performance testing will improve the performance of the site under test by enableing technical resources to adjust tuning ettings in the applciation and therefore make it as lean as possible meaning more people can access the site and have a satisfactory response from it at one time. However if the whole country suddenly demands the web site we'll probably still have a problem in these rare cases. Perhaps a solution might be to use a queueing mechanism similar to that that we see in call centres - giving everyone a chance to use the system. Either way performance testing and tuning will ensure that the application is the best ist can be.

Application level monitoring from an end user perspective would alert operatives that the site is not available from the moment that it's down, additionally as soon as the site starts to suffer with poor response times or internittent

It's on my wish list to speak to National Rail!

How to performance test a public web site.

One of the hardest things to estimate is what kind of load your web site should be expected to stand up to – especially if the web site or web application is a new one. In essence what we’re trying to do is work out non-functional requirements before running the test.

Really there should be a requirements specification that the web application was built to. Something like:
“Response time must remain below 7 seconds”
“The application must be able to handle 3,000 concurrent users.”
“The application must be able to handle 500 transactions in one hour.”
The test then has to be run with 3000 concurrent users at peak, and while at that peak should be able to complete 500 business transactions in one hour. All the time page response time should never exceed 7 sends.

But what can be done if this kind of information does not exist?

We had this challenge when a customer needed a performance test running but had no idea of how many users the site should support or how many transaction he might expect in peak times.The customer had some statistics from the TV company they were advertising the web site through. GMTV told them that after a similar campaign the TV company themselves ran they saw 25,000 unique registrations on their web site in one day.

At the time nothing was known about the performance capabilities of the web infrastructure the customer had so the following approach was taken.
1: Evaluate what the system is currently capable of
This was done using our cloud internet load testing service & it was found the application could deal with only a few hundred users while under a load of a few thousand transactions per hour, it was deemed that the application was unusable if page response time was greater than 12 seconds. The same method was repeated every time the application was tuned for performance.
2: Establish a support mechanism in case the number of concurrent users the system could support was breached
A simple queuing system was developed which in effect increased the capability of the system to handle several thousand concurrent users.
The outcome was that the applications performance testing exercise was highly valuable provide. Primarily performance testing enabled performance tuning exercises to be run making the web application as lean and as fast as possible with current infrastructure. Additionally it provided the known limits of the application enabling the metrics for the queuing system to be specified.

Performance Testing Blog

Here it begins - my performance testing blog!
I'm going to try to update this frequently - whether that's monthly (at least) or weekly yet remain s to be seen.
It should be (I hope) interesting and relevant and where I get permission from customers and partners I'll include some case studies and short stories.

So let's start with my 1st entry: What is performance testing?
We can quote ISEB’s definitions, what we find in Wikipedia or what the current buzzword using conference speaker is saying which are all valuable (perhaps not so much with the latter), but where would be the value in repeating already stated definitions? I’m going to start with what I see as the definition of performance testing & you can decide if it’s valuable or not.

Performance Testing from my experience 90% of the time refers to evaluating the performance of applications deployed via web / HTTP technology. The other 10% of cases may be deployed via other protocols such as Citrix (ICA), Oracle (OCI), Oracle Forms (OCX?), SAP (proprietary), Remote Desktop (RDP) & bespoke (usually some for of TCP / socket connection) and many more.
Either way in all cases the application whose performance is to be measured is critical to the business, in fact it’s usually a core application envisaged to be used every day at high volume. Often these are internal applications such as financial reporting software, content management systems, timesheet systems and so on. If not internal applications the company itself may in reality be the web site application, take the example of an on line betting company, or a trading company like ebay. The companies performance (in $/£/€ terms) is directly related to the performance of the web site. Alternatively the company may rely solely on the web site as it’s point of contact with the public, take an events company – the web site may be used to register athletes for a run, it may be used to sell tickets for a concert.
So if performance testing is so closely related to business performance what is it we really need to measure? Well if we go back to what business’ main goals are, they relate to transactions – business transactions. So in fact most of the time the application under test has to support a given transaction rate to be able to support the business. It’s not as simple as that though as the transaction rate has to be maintained while a certain number of real users are using the application, additionally the application has to remain stable (up and running) and usable (i.e. not tortoise / snail like).
So take a timesheet application for example, end of the week or month a global company has 10,000 employees all needing to submit timesheets. Human nature dictates that people generally leave this to the last minute. However due to time zones there is a 24 hour windows for the timesheets to be submitted and the biggest country has 4,000 employees. Worst case scenario is that at 5pm that office decides to submit all timesheets in that hour, this means the application will need to deal with 4000 timesheet submissions in 1 hour, if you have the luxury of having historic data (perhaps from a previous system) you may know that you typically see no more than 400 users accessing the system at one time.
This means we can say that we need to simulate 400 concurrent users and run the test for at least an hour at a timesheet submission rate of 10,000 per hour.
What about public web sites where access is simply not controlled – that’s pretty difficult but I’ve heard stories about how companies deal with that.
This entry’s pretty long now so I’ll talk about that next time.