Sharepoint Performance / Load Testing using Microsoft Visual Studio vs Quotium QTest

Below is a white paper written in 2009 - QTest is now even better for performance testing Share point - contact me for more information.

SharePoint Performance Testing White Paper: Scripting

Performance Test Scripts for Complex SharePoint Applications by Adam Brown

Introduction

This paper focuses on the scripting phase of performance testing as this can be one of the most time & effort intensive parts of any load testing project, particularly when working with bespoke and complex applications, including SharePoint applications. Also this can be the phase where most technical experience is required.

Abstract

The reason for the paper is following an attempt to use Microsoft Visual Studio Team Test System Web Testing to performance test a complex SharePoint implementation.

It was found that creating scripts (transactions, user journeys whatever you like to call them) for simple SharePoint applications could be straight forward, however for a more complex implementation the dynamic parameterisation that visual studio does simply did not parameterise all necessary variables and a coding approach was required. The same scenario was then evaluated using QTest from Quotium which was able to parameterise all variables quickly and without the need for coding.

Executive Summary

·  In this scenario, Visual Studio Team System for Testers would require at least 540 – 1080 lines of code to be manually edited or written plus other investigative activity to make a suite of scripts.

·  QTest was able to handle the required parameterisation without any code having to be written; QTest generated and integrated all the code required.

The application under test (AUT)

The objective of the application was to make more of the functionality of Microsoft Office Project available via a web interface, this was then deployed via SharePoint so that staff unfamiliar with MS Project would be more comfortable with a web interface and MS Project did not need to be installed on all machines. It enables users to create, edit and view projects. The application was written by Raona of Spain.

The scenario

We decided to record the process of creating a project using the application, that way we could see on the server if the transactions simulated with the tool were being properly executed. The transaction steps were as follows:

1: Navigate to Application HomePage
2: Click New Project
3: Enter Project Name, Template & and Zone  Click Create Project
4: Click Save and Publish
5: Click OK when asked to check in
6: Click Save

The captures

Captures were made using the built in recording mechanisms featured with each tool, this comprises of clicking the record button and interacting with the AUT as a real user would.

Microsoft’s Visual Studio Team System for Testers

First we used the Microsoft tool, capture was straight forward and the tool appeared to detect dynamic parameters while generating its visual script (no code seen at this point). Dynamic parameters are incredibly important when generating scripts as if these are not dealt with correctly then the entire script (and any load tests executed with it) can be rendered useless.

Looking at the script that was generated and the dynamic parameters, at first glance the tool had done a good job.

It was clear that we would have to parameterise the project name (used in step 3 of our transaction steps list above) as duplicate project names are not allowed in the AUT, however before parameterising this value we thought we’d better check that the script would run after simply manually updating the project name parameter in the script by using find and replace (we changed it from “TestProj2” to “TestProj3”). This way we could quickly find out what else, if anything needed to be parameterised.

After attempting to run the script with the new project name parameter it failed receiving a HTTP Code 500 - internal server error from the application under test.

Note the Server 500 Error (highlighted in blue) and the details in the window below it.

After a closer look at the script (at this point we dropped down into the generated Visual Basic code) we could see exactly what had been automatically parameterised and what had not and it quickly became obvious why this script had caused a 500 internal server error and why this script, in it’s current state, could never work and could never be used to generate accurate load.

The reasons for this are explained below as follows:

The dynamic parameters that the Visual Studio Web Testing tool did not deal with are as below:

ProjectUID
CLSID
SessionUID
JobUID
ViewUID
ViewTimeStamp

The problem is that the AUT as with many applications needs to use these types of parameters to maintain state, identify objects and maintain sessions. To use a parameter from a previous transaction recording simply can’t work and if it does not cause visible errors, it will result in inaccurate load being generated were the script to be used as part of a test.

The parameters it did deal with were as follows:               __VIEWSTATE
__EVENTARGUMENT
__EVENTVALIDATION
__REQUESTDIGEST
__LASTFOCUS
Cookies
URL Parameters in redirects

These are standard Microsoft parameters and are dealt with correctly. The problem with the parameters mentioned previously is that they are more than likely created in the development process, so Visual Studio’s web testing tool can’t know anything about them.

An example of a parameter that has not been parameterised is shown below is the XML request made by Visual Studio during a failed replay of the script.

If we look at the script below we can see the parameter is static in the script not dynamic (look in the chunk of unparsed XML):

Compare this with a parameter that has been automatically parameterised and made dynamic (see __VIEWSTATE):

So it seems that the only way to make the script work is to manually insert code to extract the dynamic parameters from the relevant web server responses, store them as variables and insert them in the place of the static parameters that need to be replaced in the XML.

Of course this should be no problem for a seasoned developer or tester with VB / C# coding / scripting experience, however it may be time consuming as there are at least 6 parameters here that need to be replaced, each of which appears any number of times in the script, depending on the size of the script. Add to that the fact that we will need to produce more than one script to create a realistic scenario, furthermore, when the application changes, the script will more than likely need to be re-recorded to ensure that any changes are correctly handled in the script. This makes for a lot of lines of code and hence a lot of time spent during the scripting process. Let’s quantify this:

Once we’ve figured out a way of finding out where the parameter came from and the best way to extract it with Visual studio we have the following:

6 Parameters each appearing on average 6 times in each script.

This means that approximately 36 lines of code need to be altered.

Further more at least 18 lines of code need to be inserted for declaration and extraction (1 for declaration and one for extraction – possibly more).

More lines of code may be required for complex parsing of parameters.

This means that for each script we have there are at least 54 changes required.

Typically in load testing 10-20 scripts are required for an accurate scenario, this means that we have at least 540-1080 lines of code to edit or insert for each load test we prepare.

If the application is changed then all of this work has to be re-visited or repeated.

What’s required is a parsing engine that can be configured to deal with bespoke / non standard parameters.

Quotium QTest

Next we used QTest to capture the same transaction. QTest does not automatically parameterise the script once it is generated; automatic parameterisation is achieved by selecting a model from the drop down box and clicking ‘Apply Model’. QTest is not limited to Microsoft technologies so there are other models on there for J2EE, Siebel, SAP etc.

Note the Model drop down list in the top left of the application

By default the SharePoint model was not there, this was downloaded separately as an XML file. We found however that the .Net model was able to make most of the parameterisation that Visual Studio could.

After the parameterisation process QTest had covered everything that could be seen as a header parameter, it had not covered any parameters that appeared in the body part of a request, such as an XML request, these remained static.

It was however quite straight forward to parameterise these using Find In HTTP Responses from the right click menu (see above). Highlighting the parameter we needed to make dynamic and right clicking presented the menu we needed (below):

QTest then presented us with the locations of every instance of this parameter in all of the HTTP responses (lower part of screen in screenshot above)

After double clicking the response (above: I chose the one from the body of response ID 87 rather than the header of response 87) QTest then highlighted the parameter for extraction in the HTTP window on the right of the tool (see highlighted text in widow below),

where we were then able to select Extract Value following a right click on the selected text (see right). QTest then evaluated the text to the immediate right and to the immediate left of the parameter for extraction and used this to build an extraction rule (see Left Delimiter and Right Delimiter in screen shot above).

Using the Magnifying glass button we were able to verify that the extraction had worked correctly. Finally, the Apply button created the variable in the script, generated the extraction code and inserted it at the relevant place. All that was left was to use the find and replace to replace all instances of the hard coded value in the script with the variable that had just been parameterised.

By highlighting the static value in the script, right clicking and using Find and replace in strings… it was possible to quickly parameterise all instances of this static value as per screen shot to the right.

This process was repeated for all variables that remained hard coded in the script including the ‘Project Name’ that the user would normally enter through the keyboard.

Rather than use a list of values we decided to use a timestamp on the project name as that way we would always have a unique name for the Project. Looking at the help we found the command for timestamps. This did mean that we had a small piece of code to write as follows, this was placed at the top of the script:

UniqeName = “MyProj” + Date(“%f”);

To save time we then modified the SharePoint model to ensure that all references to the hard coded project name were parameterised. To do this we selected ‘SharePoint’ from the model drop down list on the toolbar and clicked the 'Apply Model' button to edit it. We inserted a rule for the project name to ensure that the variable we just created was used instead of the hard coded value. Please see the screen shots below.

The next step was to replay the script to see if it works.

During the replay the tool shows the HTML pages that the server responds with and finally pops up with a window offering to show replay differences. This proved to be especially useful as it compares the traffic that was generated by the browser when the script was recorded with the traffic that QTest generated. Any unexpected behaviour is quickly highlighted with this tool by severity 1,2 & 3.

Looking at the screen capture below we can see that the request that had failed with Visual Studio (see 2^nd illustration on 2^nd page) has now worked with QTest. This is because all of the parameters in the XML statements have been correctly dealt with.

We can also see in the replay window in QTest that the Project has been successfully created with the unique name ‘MyProj’ and a UTC:

This can also be verified in a browser:

Summary

Visual Studio is a capable tool in the hands of developers with the necessary experience to use it and the time to correctly program it. It can be suitable for use by non programmers with some simple web applications where nothing has been bespoken (in our experience this is a rare case in large organisations).

However if the application is not a standard out of the box vanilla affair, time is limited and programmers scarce then QTest offers a better approach as it’s features make a typically difficult and lengthy task (scripting) a relatively straight forward and short one.

QTest is a very capable and powerful tool in the hands of anyone with an IT background, developer or not. Therefore it’s highly suitable for testers.