Global Quality Journal

WEB PERFORMANCE TESTING: What is the most important aspect for success?

David Alves:

When it comes to web performance testing (or performance testing in general), the usual project involves four (4) phases. In order of execution, these phases are:

• Strategy & Planning
• Data Preparation & Scripting
• System Shakeout & Execution
• Analysis & Reporting.

In addition, during various parts of these phases, there are decision made around various artifacts/tools, such as:

• An Appropriate Workload Model
• Load Generation Tools
• Monitoring Tools/Solutions

When it comes to performance testing and workload models, the mantra is “garbage in, garbage out”. Building a workload model that is representative to what you expect to happen in the real world future is key to getting meaningful results.

Also, the right load generating tool can make or break you. You need a tool that supports the appropriate technologies, and gives you enough fine grain control to meet your needs. When it comes to load generation tools, it is NOT one size fits all. Choosing the wrong load generation tools can truly hamper your efforts. While choosing the right load generation tool can truly enable you to conduct a successful performance test.

Lastly, while the tool you choose will present front-end metrics about the workload generated, such as the number of escalating concurrent users, average, minimum, and maximum transaction response times, the monitoring tools and solutions you choose will allow you to correlate this front-end data with what is measured or observed on the back-end systems. The more in-depth and low-level the metrics captured, the quicker and more clearly you will get to the bottom of any performance issues or their proper resolution (however, typically the more expensive the tool or solution). Therefore, proper selection of an appropriate monitoring tool or solution is also key to a successful performance test.

Now that we’ve discussed the tools and artifacts that contribute to a successful performance test, let’s discuss the importance of the activities in each test phase towards your successful performance test.

Strategy & Planning

During this phase an appropriate workload model is chosen, a load generation tool, and monitoring tools/solutions. This in turn dictates the script language, data preparation needs, and approach. In addition, an escalating model for increasing the workload, or changing the system configuration during the execution phase are planned in accordance with the workload model. This phase affects and dictates decisions in all other phases. Therefore, one could easily say that this is the most important aspect to making sure your performance testing is a success. However, before making this declaration, let’s give each of the other phases and artifacts an equal perusal.

Data Preparation & Scripting

For a performance test to be successful, clearly you must manage your test data. There is front-end data and back-end data. Front-end data is the data your users enter into the application under test while interacting with it. Back-end data is the stored data in application databases and other storage solutions, such as user account information, product inventory, etc. Clearly you need to have good, clean, scrubbed front-end and back-end data, as well as the processes in place to restore the back-end data to its original state after it has been modified by the front-end data applied during the test, in order to make your performance test successful. So maybe this is the key to a successful performance test?

Also however, you need to have accurate scripts, in a language that gives enough control to model the workload chosen as close as possible. This too is important for a successful performance test.

System Shakeout & Execution

System Shakeout is an activity that novices all too often fail to perform. Before running official tests, you need to make sure that your test system and your test environment, including test data, scripts, workload, etc. are all properly setup and running effectively, and as planned. All too often I’ve seen a novice only find at the very end of their testing effort that their test was compromised by an misconfigured test system, or test environmental issue.

Test execution and the various workload stepping stones that lead to the very final, full workload test, are very important. Interim test executions must be in line with the testing goals and objectives, and introduce rigor of the process and methodology in regards to test configuration modification and management, in order to gain further insight into the resolution of problems, bottlenecks, or the general scalability of the system. Cleary proper test system shakeout and execution are important to the success of any performance test.

Analysis & Reporting

Analyzing the results of all test runs, looking for data correlation between the use of various back-end system resources, workload, and response times is key. It is the whole reason we do the test in question, and how we convey the results to our stakeholders. It takes significant skill and previous experience, as “correlation” does not always mean “cause”. Root cause analysis is part art and part science, relying greatly upon past experience and exposure to different technologies, etc. Again, an activity that heavily contributes to the success of a performance test.

Conclusion

So, what is the most important aspect for a successful performance test? Is it the load tool, the monitoring tools, the strategy and planning, the analysis and reporting, or one of the other test phases?

The most important aspect or factor that truly determines the success of a performance test is the Performance Architect participating in all of the above. Being a Performance Architect requires a lot of specialized skill and experience. Performance Architects must have the rigor of process and procedure required by quality assurance and control engineers. They script in languages such as “C” or Java, and therefore must have development skills. The more years of experience, and the more technologies they have been exposed to, the better their strategy and planning, their marrying of load generation tools to application technologies, their understanding of system issues and bottlenecks, and their ability to analyze all data presented (and there is a mountain of data created by a performance test) and to make head and tails of it. While all of the phases, tools, and other activities discussed above are crucial to a successful performance test, having the right Performance Architect to guide these test phases, critical decisions, and to bring their skillset to the table; this is THE most crucial aspect of any performance test.

(David Alves is CTO of Global Quality Partners)

posted by: David Alves

Is there anything still to be learned about scalability and performance from HealthCare.gov?

Greg Margolin:

Politico.com recently reported that ever since HealthCare.gov failed to pass real life scalability test, President Obama took to a habit of knocking on wood every time he hears a bit of a good news about the site.  Indeed the term “scalability testing” made it to board room and general media discussions. Shortly after the launch, it was amazing and amusing for software quality assurance veterans to watch media pundits and industry tycoons to finally expand on the need for user acceptance and scalability testing. Aetna’s CEO, Mark Bertolini, plainly stated in a video interview that prior to the HealthCare.gov launch, it was clear that the site was not even ready to start proper scalability testing.

The situation in numbers looked this way — HealthCare.gov was designed with the intention to scale up to 50,000 concurrent visitors.  In reality the site could barely handle 10,000 concurrent visitors, while the load on the first day was around 200,000 concurrent visitors.  So, what is scalability?  The definition states that “Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth”.  In other words, scalability means the ability of the system (and system design) to handle an ever growing work load by adding appropriate resources.  Scalability could be horizontal, as when additional nodes are added to the system, or vertical, as when the central node is being beefed up with additional resources.  When the system cannot effectively handle more work, despite available resources, this means that the system hit its scalability ceiling.  In software system engineering, scalability belongs to a group of awkwardly named quality attributes known as "non-functional requirements".  While this group of requirements is defined by what they are not — akin to "nontechnical staff" or "wireless communication" — it has been proven to be absolutely vital for today’s IT systems.  At the time of the HealthCare.gov launch, industry experts stated that problems with the site were not unique, just the opposite, they were all too common.  In today's fast pace world, software development teams are all too often solely focused on launching a site by a certain date, and on making sure that core features work.  Scalability (and often) security fall by the way side.  Then, immediately after launch, a rude reawakening comes to light, when what should have been a welcome success — more concurrent users than originally planned — turns into a disaster capable of killing a company.  The story of HealthCare.gov has a happy ending so far.  The best engineers from American leading companies descended on the site and feverishly worked on making it scalable — by partitioning the site, moving it to the cloud, prioritizing work load, and testing architectural changes in the process under real life and simulated loads.  So, today, six months and one Secretary of Health and Human Services later, HealthCare.gov is holding its own under current load.  There are applicable and non-applicable lessons for Main Street businesses in this story.  What’s not applicable?  Not every Joe can call Google, Amazon, and Verizon, etc. and ask for immediate access to the best industry experts.  Not every business can as easily sustain a sharp expansion in the budget allocated to their site the way federal government can. Does it mean that for Mr. and Mrs. Main Street’s e-Commerce web site there is no hope?  Not at all.  Thanks to open source software, everybody can have access to many of the tools used by industry giants.  Google in its performance and scalability testing heavily relies on JMeter (an open source tool) and did a lot to make the tool much more robust and full-featured.  The proliferation of cloud environments, pioneered by Amazon, allows for rapid deployment of test tools in the cloud and the creation of test environments on demand.  Finally, there are also industry experts such as Global Quality Partners, LLC available to put all the necessary components in a properly packaged, affordable, scalability and performance testing service.

The lessons of HealthCare.gov are there for all of us in plain view. It is for us to learn from them, or to ignore them.  Modern eCommerce sites should be designed with scalability in mind. For scalability testing there is JMeter and the cloud.  And, of course, there is always knocking on wood.  Just ask the President.

(Greg Margolin is a Managing Partner of Global Quality Partners)

posted by: Greg Margolin