Web applications Performance Testing Metrics

Introduction

Few years back the use of World Wide Web (WWW) was limited to services industry but today this is web era where everything is over the web. Now organizations are not only providing online services to their clients but also using this platform to completely run their businesses. This excess of use and importance of online services also greatly demand for their hassle free delivery. These days’ users are very demanding and user abandonment starts as they received response time greater than 3 seconds. Slow performing applications have direct impact on business performance and according to a study $45 billion business loss occur yearly due to poorly performing web applications.

Performance testing is conducted to make sure the application under test (AUT) will meet user expectations in production environment. Based on the results of the performance testing activity, required actions are taken to fill the gap between the actual and expected results. As effective measurement is the key for the success of any process, similarly performance testing activity required results can never be achieved without having clearly defined success criteria. Different performance testing metrics are used to measure the output of the performance activity. These performance metrics provides the quantitative method to evaluate the test results. Information obtain from these metrics assist organizations to improve their productivity, drop error rate and improve product quality and services level to achieve their desired business goals.        

There are dozens of performance counters which are generated during the performance test and it’s almost impossible to thoroughly evaluate each of them. Being a good performance engineer, one must have strong understanding of performance problematic areas of the AUT and right set of performance counters of each of these areas. In this discussion, I will highlight different aspects of performance metrics, potential performance problematic areas of web applications (Clint side, Web Server, Application Server, Database Server and Network Configurations) and performance counters of each of these areas in details which need to be monitored during performance testing.     

What is Metric?

Metric is a standard unit of measurement which calculates the results. Software Metrics are used for the evaluation of software product and its services.

Performance testing metrics are used to evaluate the application performance parameters and to identify which areas of the application are creating performance bottlenecks.  

Importance of Metrics

Quantitative results always provide the best outcome of an activity and metrics help in getting these quantitative results. Following points also highlight the importance of performance testing metrics,

• Metrics are used to improve the product quality to achieve better customer satisfaction
• Metrics provide easy and clear output of the activity and identify areas which need more attention
• Metrics help in comparing the results of different test to find out the impact of any change in application
• Metrics are monitors which provide the exact cause of problem
• Metrics establish baseline for all tests
• Metrics track project progress

Metric Characteristics

There are dozens of metrics generated in performance test but it’s not possible to evaluate all of them. Specific performance metrics are selected based on following characteristics,

• Metrics whose complete and accurate data can be collected should be evaluated
• Metrics which are important in the context of the application should be evaluated
• Metrics which are easy to calculate should be evaluated

Performance Testing Metrics selection

It’s normally depends upon the AUT performance goals to decide which software metrics should be evaluated. But there are few generic metrics which are normally considered in every performance test as listed below.

Performance Test Requirement Gathering Metrics

The scope of test and its activities always depends upon the requirements. Having crystal clear requirements before starting the performance testing activity is a fundamental step. A test with vague requirements can never achieve its desired results. So in first phase being a performance engineer, one need to identify the set of metrics required for gathering complete list of performance requirements. Different stakeholders can be contacted to collect performance requirements but viewing the server log is also helpful activity. Following server side metrics are considered for a live application to establish performance goals,

• Total Page Views per Week
• Total Hits per Week
• Total User Sessions per Week
• Average Page Size
• Average Hit Size
• Page Request Distribution
• User Abandonment

Total Page Views per Week

Page view is the request on server for that specific page with all its embedded objects. This metric provide the information about the weekly traffic on AUT. Choosing the page views metric over page hits or byte transferred is preferred because it’s commonly used as an indicator for website traffic. Moreover, choosing the weekly limit also provides more realistic application traffic analysis over choosing the specific hours and day traffic.

Total Hits per Week

A hit is any resource (web page, image, files etc.) request received by the web server from the client. Several hits are made on server when client request for a web page. Web pages normally made of number of images and files and number of hits to web server for a specific web page will be equal to the number of resources it contains.

Total User Sessions per Week

A user session is a unique user visit on the website. This user uniqueness is maintained with many different approaches like with username and its password, browser cookies and the user machine IP address. Tracking user session is very handy in load testing because it’s not only provide the information of number of users accessing the application but also provide the user navigation trends on the application which is very important to simulate real user load during testing.   

Average Hit Size

This is the average amount of data user received from the web server against a particular hit. Average hit size is measured in Kbytes.

Page Request Distribution

Page request distribution metric represent the user request distribution in percentage across all the website pages. This metric provides useful information on user trends and help out in deciding the user distribution on performance schedule. 

User Abandonment

This metric provides the information on amount of time an average user waits for a page load before exiting from the application in dissatisfied manner. This value helps in deciding about the user acceptable response time limit.

Client-Side Requirement Gathering Metrics

End users who interact and use the application to fulfill their specific requirements are called clients. The usage patterns and the application behavior to all these clients can vary greatly depending upon different variables which are called client-side variables and simulating all user requirements is necessary for designing effective performance test. Following client side metrics should be considered while designing a performance test,

• Interaction Speed
• Latency Tolerance
• Familiarity
• Connection speed
• Location

Interaction Speed

Interaction speed represents the user interaction speed with the applications. This variable represents the how fast a user perform business actions on a web page and navigates between different web pages.

Latency Tolerance

This variable provide the information of how much a user waits for page response before taking next action which could be application abandonment, page reload etc. Today on average a user waits for 3 seconds for a webpage to load before taking any action.

Familiarity

This variable represents how much user is familiar with the application. Frequent users are always able to complete the required task with much ease in much less time as compared to new visitors.

There are few client-system variables as well which needs to be considered before starting the load testing,

Connection speed

It’s important to know what percentage of users is using which internet connection. Application response time will be much lower on fast internet connection as compared to lower connection. Moreover for realistic performance test, this internet connection bandwidth should be considered for virtual users as well.

User Geographical Location

User geographical location also greatly affects the user experience on the application. Application response time will be less for lesser number of hops between the client and the server.

We have discussed the requirement gathering metrics and now need to discuss the performance metrics evaluated during the performance test. Performance testing metrics can be divided into following broad categories,

• Vuser Monitoring
• Transaction Monitoring
• Web Resource Metrics
• System Resource Monitoring
• Web Server Performance Metrics
• Application Server Performance Metrics
• Database Server
• Network Monitors
• Client Side Metrics

Vuser Monitoring

It’s very important to know the status of every user on every executed transaction to analyze the impact of increasing workload on the system. There are various graphs available in every performance testing tool for monitoring the Vusers activities and some of the most important are as following,

Running Vusers Graph

Running Vusers metric provides the complete information of how many users are running, how many are waiting for their turn and how have executed at any particular time of the test. It provides the information of number of users currently accessing the application.

Vusers with Errors Graph

One of the most important factors while analyzing the executed performance test results is to know how many users were unable to complete their actions and which errors they faced. Knowing this information not only provides the application performance insights but also help in identifying the problem root cause.

Transaction Monitoring

Transactions are business processes which are executed for certain user to test the application performance. It’s vital to know the status of each executed transaction and its response time. Following are some of the important metrics which should be carefully monitored during the performance test,

Average Transaction Response Time 

There is no doubt that response time is the most user concerned performance parameter. An application or transaction with slow response time will never be accepted to users. So each executed transaction response time is monitored during every second of the scenario run to evaluate the running users’ impact on performance. These days average transaction response should not be more 3 seconds otherwise user abandonment started.

Passed/Failed Transactions

Sometimes (especially for business critical applications) it’s more important to complete the certain business transaction rather than their slow response time. So knowing the information of how many transactions passed and failed during the performance test is also extremely important.

Web Resource Metrics

Web resource metrics provide the web server parameters information captured during the performance test. Following are few important web server metrics which should be monitored during the performance test,

Hits per Second

Hits per second graph provide the information of number of HTTP requests sent to the web server during specific time period of the performance test. This is one of the most important metric which shows how the application behaved on increasing the user load. Hits per second increased by increasing the user load unless the application is able to handle that specific user load.

Throughput

Throughput graph provide the information on server response during specific time period of the performance test. This is also extremely important metric as it provides the information of web server response against the user requests. Throughput value is also directly proportional to user load unless web server is successfully able to handle that specific user load under those conditions.

HTTP Responses per Seconds

HTTP responses graph provide the information of all the HTTP status codes generated per second during the performance test execution. These values provide useful information on web server state during every second of the performance test. Following is the summery of response codes,

1XX: Informational (Client request is received)

2XX: Success (Client request is received, accepted and processed successfully)

3XX: Redirection (Client needs to take extra actions to complete the request)

4XX: Client Error (Error in client request)

5XX: Server Error (Server failed to complete the request)

Errors per Second

A poorly performing application not only takes more time in completing its business transaction but also generate different types of errors as well. These errors stopped users to complete their transactions. All errors don’t have the same impact on the AUT. The seriousness of the error can be analyzed from its HTTP response code as mentioned above.

Pages Downloaded per Second

This graph displays the information of web pages downloaded in every second of the test from the web server. This metric provides the useful information on amount of load generated by virtual users on the web server in terms of web pages.

Connections per Second

Information on number of new TCP/IP connections opened and shut down during each second of the performance test is provided with connections per second graph. Connections opened/shut down should be smaller than the hits per second as TCP/IP connections are very expensive. To the best interests of the system, several HTTP request should use a same TCP/IP connection.

System Resource Monitoring

Monitoring all the machines (Servers and load injectors) and their operating system resources which are being used in performance testing is also provides useful insights in analysis. Modern performance testing tools provide the facility to monitor resources on all machines being used during the test to pin point the performance bottleneck root cause. Following metrics are usually considered across all systems while a performance scenario is running,

CPU Usage

This metric provides information on machine CPU utilization. For effective application performance CPU utilization should not be more than 70%.  

Memory Usage

Memory usage provides the information on amount of free disk space, virtual memory pages moved between main memory and disk storage and percent memory used during the scenario execution.

Disk Space Usage

This metric provides the information on percent disk space is used.

Service Monitors

It provides the information on running processes on the machine and the amount of system resources they are using. The problematic processes which use more resources can be identified and fixed quickly from these monitors.          

Web Server Performance Metrics

Web server metrics provides the information of resources usage on web server during the performance test. These metrics provides the useful information on web server performance and its bottlenecks. There is long list of metrics which are generated during the performance depending upon the type of web server. Apache HTTP, Microsoft Internet Information Services (Microsoft IIS), Lighttpd, Sun Java System and Jigsaw server are famous web servers. As I mentioned above, list of performance metrics monitored during performance test varies with the type of web server, following is the list of performance counters to need to be monitored for Microsoft IIS web server during performance testing.

• Bytes Sent per Second
• Bytes Received per Second
• Get Requests per Second
• Post Requests per Second
• Maximum Connections
• Not Found (404) Error per Second
• Private Bytes
• Percent of Processor Time
• System Processor Queue Length
• Percent Disk Time
• Queue Length
• Cache Hit Percent
• Request Wait Time
• Request Queued
• Transactions per Second

Application Server Performance Metrics

Most of the application computational activities are performed on its application server. We can say application server is the back bone of the application especially in case of highly complex business applications. Web Sphere, JBoss, Apache Tomcat, Web Logic, Microsoft .Net, and Oracle Application server are some of the famous applications servers being used these days.  

Every type of application server has its performance counters based on its model. Following I am listening some of the Web Sphere application performance counters which needs to be monitored during the performance test,

• Memory Free
• Total Memory
• Memory Use
• Bean Destroys
• Active Threads
• Total Threads
• Percent Time Maxed
• Connection Wait Time
• Connection Time
• Connection Percent Used
• Active Transactions
• Transactions Suspended
• Rolled Back
• Timeouts
• Servlet Errors
• Sessions Invalidated

Data Base Server

All the application data is stored in database and the system on which database is installed is called data base server. As the database server contain all the critical information of the application, its fast and error free working is extremely important. Oracle, DB2, Microsoft SQL Server and Informix are famous database servers. Some of the common metrics monitored for all database servers are following,

• Database Transaction Summary: Database transaction monitor provide the information of volume of data sent and received by the database server. It provides the detail in bytes of requests sent and received during the performance test.
• Database Connection Summary: This metric provides the information of total number of open and close connections on database server during the test. Excess of database connections can badly affect the database server performance.
• Database Thread Summary: Thread metric provides the information of number of new threads created, connected and used.
• Database Query Summary: Read, write and delete data are main database functions and their detail is very important to check the database server performance during the performance test. Database Query metric provides the information of total number of queries read, wrote and deleted by the database server.

Some of the most important performance counters of Microsoft SQL Server are following,

• Transactions per Second
• Log Cache Hit Ratio
• Log Cache Reads per Second
• Log Bytes Flushed per Second
• Log Flush Wait Time
• Log Flush Waits per Second
• Log Flushes per Second
• Percent Log Used

Network Configurations Metrics

Network configurations play a vital role in application performance. An application can never perform well if its network is poorly designed. Each application network consists of multiple segments and it’s very important to properly monitor each and every network segment delay time during the performance test. All Modern tools are equipped with the facility to configure the network monitors in performance test environment to see the impact of network configurations on application performance.

Some of the important network metrics within a network segment which needs to be monitored during the performance test are following,

• Bytes per Second Sent to Network Interface Card
• Bytes per Second Received by NIC
• TCP Segments Sent per Second
• TCP Segments Received per Second
• TCP Segments per Second
• TCP Segments Retransmitted per Second
• TCP Connections Failures
• TCP Connections Reset
• TCP Connections Established

Client Side Performance Metrics

According to a Yahoo study, 80-90% of the web page load time is spent on front-end and its application performance can be improved up to 40% by cutting half of the front end load time. Performance engineers are keenly focusing the front end usage of resources as users are demanding better applications performance. Good performance testing tools provide the details of time consumed on various web pages resources like images, JavaScript files, CSS, html, DNS connection, TCP connection, SSL handshake, HTTP Request, HTTP Response status and response size. Some of the client side metrics need to be monitored during performance testing are,

• Images Load Time
• JavaScript Files Load Time
• CSS Files Load Time
• HTML Resources Load Time
• DNS Lookup Time
• TCP Connection Time
• SSL Handshake Time
• Time to First Byte
• Content Download Time
• Content Caching Policy
• Availability Content Delivery Network (CDN)
• Resources Rendering Time
• Number of HTTP Requests
• HTTP Response Status
• HTTP Response Size

Conclusion

Quantitative results always provide the clear picture of a situation and keep track of changes. Performance testing metrics provides quantitative results which helps performance engineers in test analysis. Specific Vuser, Transaction, Web Server, Application Server, Database Server, Network Configurations and Client Side metrics are need to be monitored during the performance test to determine the application current performance and to find out its problematic areas and their root causes. 

Source : http://www.agileload.com/agileload/blog/2013/02/18/web-applications-performance-testing-metrics