Multivariate testing

Multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests performed on one page at the same time. A/B tests are usually performed to determine the better of two content variations, multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.

Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improvement in the defined goals of a website, whether that be user registrations or successful completion of a checkout process (that is, conversion rate). Dramatic increases can be seen through testing different copy text, form layouts and even landing page images and background colours. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest increase in conversions.

Testing can be carried out on a dynamically generated website by setting up the server to display the different variations of content in equal proportions to incoming visitors. Statistics on how each visitor went on to behave after seeing the content under test must then be gathered and presented. Outsourced services can also be used to provide multivariate testing on websites with minor changes to page coding. These services insert their content to predefined areas of a site and monitor user behavior.

In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of them proceeding to a defined goal. The testing is transparent to the visitor with all commercial solutions capable of ensuring that each visitor is shown the same content on every visit.

System Test - Capacity Testing

Scheduling Capacity Testing
Performance Test and Stress/Load Test occur during System Test or when enough of the application has been delivered. The earlier capacity testing can be applied, the earlier defects can be detected and mediated. It is critical to detect capacity related defects early because these types of defects often require architectural changes to the system. Because these tests usually depend on functional interfaces, it may be wise to delay until function testing has demonstrated some predefined level of reliability. A balance must be struck between testing early and testing when appropriate.

Designing Capacity Tests
Lack of capacity testing requirements and goals is one of the most common errors made during any capacity testing exercise. Test organizations will often go through the process of capacity testing only to discover the testing results do not present the project with any useful information. The solution is to treat Performance and Stress/Load testing the same as any other testing effort. The test organization needs to perform: Test Planning, Partitioning / Functional Decomposition, Requirements Definition / Verification, Test Case Design, Traceability (Traceability Matrix), Test Case Execution, Defect Management, and Coverage Analysis (for more on this see "Requirements based Function Test").

Implementing Capacity Tests
Both Performance and Stress/Load testing require a toolset that can put the system under a known load and measure the performance of the application while under load. Several shops have developed their own in-shop solutions for capacity testing and there are several capacity testing freeware, shareware, and commercial products available to meet this need. It is easy to fall into the .over-engineering. trap when implementing a capacity testing toolset - to avoid this trap ensure the solution meets the test organizations goals and that the toolset does not become a "goal" in itself.