Market Research Techniques 2 min read Updated June 30, 2026

A/B Testing

A/B testing is an experimental research method that compares two versions of a product, we…

A/B Testing — Definition

A/B testing is an experimental research method that compares two versions of a product, webpage, or marketing asset (version A and version B) by showing each to a random sample of users to determine which performs better against a defined metric.

Key Takeaways
  • A/B testing requires randomly assigning users to control (A) and variant (B) groups
  • Statistical significance (typically 95% confidence) is required before declaring a winner
  • Tests should run for a full business cycle to control for day-of-week effects
  • Only one variable should change between A and B to isolate causation
  • Sample size and minimum detectable effect must be calculated before launching a test
Advantages
  • Measures actual behavior rather than stated preference
  • Provides statistically rigorous, causal evidence of impact
  • Scales well in digital environments with large traffic volumes
  • Results are directly actionable — implement the winning variant
  • Continuous testing builds a culture of evidence-based decision-making
Limitations
  • Requires sufficient traffic/sample size to reach statistical power
  • Can only test one or a few variables at a time without complex multivariate design
  • External factors (seasonality, promotions) can confound results
  • Stopping tests early based on interim results inflates false positives
  • Findings may not generalize beyond the specific tested context

How A/B Testing Works

A/B testing (also called split testing) randomly divides users into two groups: the control group sees the existing version (A), while the variant group sees the modified version (B). Performance is measured against a predefined success metric — conversion rate, click-through rate, revenue per visitor, or similar.

The A/B Testing Process

  1. Form a hypothesis: "Changing the CTA button color from blue to orange will increase conversion rate"
  2. Define success metrics: Primary metric (conversion rate) plus guardrail metrics (bounce rate, time on page)
  3. Calculate required sample size: Based on baseline conversion rate, minimum detectable effect, and desired statistical power
  4. Run the test: Randomly assign traffic, typically for at least one full week to control for day-of-week patterns
  5. Analyze results: Check for statistical significance (typically p 0.05) before declaring a winner

A/B Testing vs Conjoint Analysis

A/B testing measures real behavior with two specific variants in a live environment — high external validity but limited to testing one change at a time. Conjoint analysis measures stated preferences across many attribute combinations simultaneously in a survey environment — broader insight but stated rather than revealed preference.

Common A/B Testing Mistakes

  • Stopping tests too early: Checking results daily and stopping as soon as significance appears inflates false positive rates ("peeking problem")
  • Testing too many variables at once: Makes it impossible to know which change drove the result
  • Insufficient sample size: Underpowered tests produce inconclusive or misleading results
  • Ignoring seasonality: Running a test during an atypical period (holidays, promotions) skews results

Frequently Asked Questions

What is statistical significance in A/B testing?

Statistical significance (typically p 0.05, or 95% confidence) means there is less than a 5% probability that the observed difference between A and B occurred by random chance alone.

How long should an A/B test run?

Minimum one full week to capture day-of-week variation, often 2-4 weeks to reach adequate sample size and account for any novelty effects wearing off.

Ambarish Kumar Verma
Ambarish Kumar Verma
Founder, MarketResearchReports.com · 17+ years in Market Research

Ambarish has been writing about market research since 2012. He is the founder of MarketResearchReports.com, a leading market research platform.