WEB PERFORMANCE TESTING: What is the most important aspect for success?

David Alves:

When it comes to web performance testing (or performance testing in general), the usual project involves four (4) phases. In order of execution, these phases are:

• Strategy & Planning
• Data Preparation & Scripting
• System Shakeout & Execution
• Analysis & Reporting.

In addition, during various parts of these phases, there are decision made around various artifacts/tools, such as:

• An Appropriate Workload Model
• Load Generation Tools
• Monitoring Tools/Solutions

When it comes to performance testing and workload models, the mantra is “garbage in, garbage out”. Building a workload model that is representative to what you expect to happen in the real world future is key to getting meaningful results.

Also, the right load generating tool can make or break you. You need a tool that supports the appropriate technologies, and gives you enough fine grain control to meet your needs. When it comes to load generation tools, it is NOT one size fits all. Choosing the wrong load generation tools can truly hamper your efforts. While choosing the right load generation tool can truly enable you to conduct a successful performance test.

Lastly, while the tool you choose will present front-end metrics about the workload generated, such as the number of escalating concurrent users, average, minimum, and maximum transaction response times, the monitoring tools and solutions you choose will allow you to correlate this front-end data with what is measured or observed on the back-end systems. The more in-depth and low-level the metrics captured, the quicker and more clearly you will get to the bottom of any performance issues or their proper resolution (however, typically the more expensive the tool or solution). Therefore, proper selection of an appropriate monitoring tool or solution is also key to a successful performance test.

Now that we’ve discussed the tools and artifacts that contribute to a successful performance test, let’s discuss the importance of the activities in each test phase towards your successful performance test.

Strategy & Planning

During this phase an appropriate workload model is chosen, a load generation tool, and monitoring tools/solutions. This in turn dictates the script language, data preparation needs, and approach. In addition, an escalating model for increasing the workload, or changing the system configuration during the execution phase are planned in accordance with the workload model. This phase affects and dictates decisions in all other phases. Therefore, one could easily say that this is the most important aspect to making sure your performance testing is a success. However, before making this declaration, let’s give each of the other phases and artifacts an equal perusal.

Data Preparation & Scripting

For a performance test to be successful, clearly you must manage your test data. There is front-end data and back-end data. Front-end data is the data your users enter into the application under test while interacting with it. Back-end data is the stored data in application databases and other storage solutions, such as user account information, product inventory, etc. Clearly you need to have good, clean, scrubbed front-end and back-end data, as well as the processes in place to restore the back-end data to its original state after it has been modified by the front-end data applied during the test, in order to make your performance test successful. So maybe this is the key to a successful performance test?

Also however, you need to have accurate scripts, in a language that gives enough control to model the workload chosen as close as possible. This too is important for a successful performance test.

System Shakeout & Execution

System Shakeout is an activity that novices all too often fail to perform. Before running official tests, you need to make sure that your test system and your test environment, including test data, scripts, workload, etc. are all properly setup and running effectively, and as planned. All too often I’ve seen a novice only find at the very end of their testing effort that their test was compromised by an misconfigured test system, or test environmental issue.

Test execution and the various workload stepping stones that lead to the very final, full workload test, are very important. Interim test executions must be in line with the testing goals and objectives, and introduce rigor of the process and methodology in regards to test configuration modification and management, in order to gain further insight into the resolution of problems, bottlenecks, or the general scalability of the system. Cleary proper test system shakeout and execution are important to the success of any performance test.

Analysis & Reporting

Analyzing the results of all test runs, looking for data correlation between the use of various back-end system resources, workload, and response times is key. It is the whole reason we do the test in question, and how we convey the results to our stakeholders. It takes significant skill and previous experience, as “correlation” does not always mean “cause”. Root cause analysis is part art and part science, relying greatly upon past experience and exposure to different technologies, etc. Again, an activity that heavily contributes to the success of a performance test.

Conclusion

So, what is the most important aspect for a successful performance test? Is it the load tool, the monitoring tools, the strategy and planning, the analysis and reporting, or one of the other test phases?

The most important aspect or factor that truly determines the success of a performance test is the Performance Architect participating in all of the above. Being a Performance Architect requires a lot of specialized skill and experience. Performance Architects must have the rigor of process and procedure required by quality assurance and control engineers. They script in languages such as “C” or Java, and therefore must have development skills. The more years of experience, and the more technologies they have been exposed to, the better their strategy and planning, their marrying of load generation tools to application technologies, their understanding of system issues and bottlenecks, and their ability to analyze all data presented (and there is a mountain of data created by a performance test) and to make head and tails of it. While all of the phases, tools, and other activities discussed above are crucial to a successful performance test, having the right Performance Architect to guide these test phases, critical decisions, and to bring their skillset to the table; this is THE most crucial aspect of any performance test.

(David Alves is CTO of Global Quality Partners)

posted by: David Alves

Comments

[Vijayanathan_N]

Great Article David. You are spot on, in saying that performance architect is the key to the success of performance testing projects. Often, we have heard of performance projects fail inspite of having jazzy tools, skilled performance test scripting teams in place for the engagement. In addition to what you have said, I would emphasize on skills of the performance test teams especially ones related to monitoring, bottleneck identification and results reporting.