During this last week, we saw many South African sites and retailers struggle to deal with the increased load experienced on Black Friday. Something which showcased that performance, load and stress testing in the country has a lot to be desired before we can compete at any international stage. Now, to be fair, I think few would’ve predicted just how fast South Africa’s adoption of a largely American shopping trend would be, which has seen retail activity on the day skyrocket exponentially from just a few years ago, so it’s easy to understand how many sites were under-prepared for the day. Still, there were many failures which could’ve been avoided if companies had tested sites properly for performance and load and been able to better scale and throttle their sites appropriately before they came down.
Now in this article, I’m not going to talk about the basics of performance testing, as I’m assuming all companies with high load variance on their sites/applications already understand that and are implementing it in their software development. Rather I want to focus on some of the extra things you can do when it comes to performance testing. Perhaps I’ll cover a more basic overview of the topic in a later article.
Below are some of my big tips for companies to follow when looking to improve their existing performance testing coverage, especially in preparation for a large-scale event such as Black Friday.
1. Monitor/ investigate user patterns
I guess the first place to start is ensuring that you understand customer usage of your particular sites or application. Hopefully, you are already tracking and keeping logs of exactly what customers are doing on the sites, which will help you to better understand their typical usage patterns and be able to answer the following questions, which all form the main focus of your specific performance tests.
What is the most commonly used part of the system?
What is the most used section of a screen/page?
Do the users require a different layout of controls to use the system more optimally?
Where does the system crash the most?
How does your responsive website render on different devices?
Where does the system respond with the slowest speed?
How long does it take to load data into the system?
How long does it take to load a page into a site?
How does the latest deployment/ build compare with previous build responses?
Does the system behave differently or slower while backend performances are occurring?
2. Test functional scenarios with different connection qualities/ Virtual networks simulation
Sending traffic over different Internet backbones and testing your app with different connection qualities and types, cellular operators and locations, will give you the confidence that the application performs consistently and reliably across a spectrum of infrastructures. Additionally, you get to see the app performing when the user changes from an open network e.g. 3G/4G to a private network e.g. secure WiFi or LAN.
3. Test in production environment
Variables such as firewalls, specific load balancing configurations, server’s memory configuration, etc., may severely influence on performance. It is best to test in production or simulated production. This will need to be done in a non-destructive manner. But the principle is consistent, test everything both physical, hardware and network to a full picture of what happens when the app/site is launched and used. Having a simulated production environment might sound like overkill, but if your business relies on performance, it’s a cost which is certainly worthwhile if you want to ensure stability at all times.
4. Test while load is occurring/ The Importance of Hybrid Load
Imagine that a functional defect can only occur under a specific load; it would be extremely difficult to find and reproduce that defect. Functional testing during protocol level load testing (hybrid load) also helps to measure and assess the user experience during different load profiles. The load scenarios are functional in their essence but they are not checked on the GUI level and don’t vary. As such the complementary functional testing that is suggested here is not limited to the scenarios that are simulated in the load. Again the move is to test as much performance as you can.
5. Evolution of Performance testing to System Performance Monitoring
Leverage your performance scripts to run constantly. With proper correlations, you will be able to drive business intelligence, such as the impact of poor or good performance on revenues and the impact of different usage scenarios on your system. The reports you will get from your load testing tool will help you to gain an understanding of the important questions, such as “what, when, why”. What this information does is allow you to plan and prioritize potential problem scenarios in the future and also better understand performance impact when your system needs to scale. Not all performance needs are equal and in times of crunch, you might add further value by only focusing on the most important problem at hand.
6. Embed performance testing into CI
Once you are comfortable with your performance, stress and load testing and that your system can meet both the demands of intense load and sustained performance you now need to identify a subset of tests that readily executed every single day. This can consist of high priority use cases and quick to execute performance tests that will prevent performance leaks entering your system at any time. While it’s a little difficult to include a sizeable production simulation into your continuous integration process, it is certainly possible to cover a wide variety of scenarios.
7. Backend Database Testing/ Monitoring
If you ignore the performance of the database that drives your application, your end user's experience will suffer. Setting and monitoring performance baselines for your database (as a standalone unit under test and as part of the system) through roll out is absolutely essential. This may include running production system stress tests to ensure your database can handle the new data loads, setting thresholds to avoid inefficient or poorly performing queries, and tracking real user response times to ensure a consistent user experience throughout the roll-out process. It’s obvious to take an end-to-end approach but spelling out the consistent elements that need to be testing will ensure you have covered all of the bases.
8. Perform Log Analysis – App and Client Side
Log files contain valuable information on the performance of your application. Not all problems are visible to users, and not all users have the ability to understand the root-cause of the problems, especially the non-reproducible. Log Analysis tools let you monitor servers and services, generate usage and error reports on the client side, build compliance analytics and more. Collecting log data from servers, networks, devices and security systems are all part of giving you as full a picture as possible of what is going on when an application is launched and used. Log analysis should be deployed during development, functional testing, load testing and production. Better to get a comprehensive understanding of error logs before users get them and stop using an application that you have spent so long (and much) developing.
As mentioned, there is more to performance and load testing than this, but hopefully, these steps will help you identify ways you could improve as a company and team.