Should you automate your tests?

Most IT Managers behave as external software engineering companies with their own organizations.

How the IT department gets involved within the company’s business issues is fundamental. Indeed they can be involved in many tasks such as:
- Defining the business needs
- Developing the applications
- Choosing the appropriate solutions on the market
- Setting and managing appropriate infrastructures
- Managing application & infrastructure resources everyday

What’s more the SLA’s between the IT department and the business departments strengthen this role of internal provider of services.

Once an application is developed or acquired it is obvious that its functionalities and its performance must be qualified and validated.

This document deals with the relevance to automate – or not – functional tests and performance tests in this context. It also deals with the application & infrastructure monitoring during the production cycle.

Functional Tests

Your goal is to check that the application meets the specified requirements. Then throughout the application lifecycle the regression tests aim at validating that the application still runs despite its evolutions. Beyond the unit tests achieved by the developers or the project managers, an independent department dedicated to validation / qualification / testing should certify that the applications run properly and meet the requirements specified.

So the first question is: Should you automate those functional tests or not? The second question: Will you be more productive so that your initial investment in functional test automation is justified? Everybody knows that automatic tests are 5 to 7 times more expensive than manual tests. They can be even more expensive when the first testing campaigns are realised by teams with poor experience in automation tools whereas a team dedicated to validation & qualification with a good experience in programming could reduce the costs.

Thus this investment will be recouped more or less quickly depending on the frequency of the minor changes included in the maintenance of the applications. The word “minor” is extremely important, indeed any major change to the application architecture or the GUI can impact the test scripts considerably. Sometimes you even have to rewrite them all.

Several contexts fully justify functional test automation. For instance software editors have to maintain several versions of their products at the same time, and this in several languages, for several system environments, for several browsers. Moreover they use several patches everyday. In this case test automation is vital – it is an integral part of the lifecycle of the products.

In the context of a specific application developed internally, it is obvious that you have to compare the initial costs – acquiring solutions, training testers and creating scripts of course – with the benefits of your test automation given the frequency of the tests made on minor changes.

Performance Tests

In this case your goal is to check that the response times experienced by end-users meet the specified requirements. Respecting those requirements is vital when the response times have a strong impact on the business goals or the user’s work. For instance if your application permits to place an order online, the global order management can take more time because of initial bad response times, and the sales figures can be reduced because the data entry component is unavailable.

Whereas it’s easy to make functional tests manually, you hardly can make load tests manually. You would have to ask dozens or hundreds of end-users to stop their work to make the tests, they would have to be synchronised on a common clock, to make the same tasks at the same time, such as to click or to type in data. Each task should be repeated whenever a correction or a change would be done.

This approach asks such a lot of energy and resources that it is incompatible with most companies.

Several solutions are offered on the market, they are absolutely reliable and they let you obtain the same results without requiring real end-users. They permit you to create scripts in half a day if you work in HTTP environments.

The cost of load test automation is 10 to 30 times less important than the cost of functional test automation. It is all the more recouped rapidly that it permits to test applications but also to validate infrastructure changes – server, network, RDBM S, system versions, browsers, etc…

Application monitoring during the production cycle

For business teams the quality of the services delivered by the IT department can be measured through 3 main criteria:
• Availability: is the application available?
• Response times: do the response times fulfil the business requirements? As outlined above response times can have a strong impact on the productivity and even on the sales figures.
• Accuracy: the end-user needs quick responses, and above all accurate responses, not “404 page not found” results for instance.

An application – when it is delivered to the business teams - implies that complex logistics is set up including most of the time: application server, database server, web server, router, firewall, provider, etc…When only one component of this chain is defective, even if the problem is local, it implies that at least one of the criteria seen above is not respected. From the IT Department’s viewpoint everything seems to work, but the end-user does not agree.

As a consequence you must check that the components of your applications are ok from both the infrastructure viewpoint and the end-user viewpoint. What’s more this correlating approach will permit you to react before the business departments suffer problems sometimes. At last a personalized map will show the deterioration processes thus it will allow locating them geographically and alert the concerned teams immediately.

by Daniel Melloul, Quotium