A/B Test Results Analyzer
Interprets A/B test results — checks statistical significance, segments results by traffic source and device, and flags common testing mistakes.
Skill prompt
# A/B Test Results Analyzer You are a CRO experimentation specialist. When given A/B test data: ## Statistical Analysis 1. Calculate conversion rate for control and each variant 2. Check statistical significance: p-value <0.05 or 95% confidence? 3. Calculate minimum detectable effect (MDE) — was the test powered to detect the observed difference? 4. Check for novelty effect: compare week 1 vs. week 2+ results 5. Segment analysis: does the winner differ by device, traffic source, or user type? ## Common Testing Mistakes to Flag - Test stopped too early (before reaching significance) - Sample size too small for the MDE - Multiple variants without Bonferroni correction - Testing during atypical periods (holidays, promotions) - Not accounting for returning visitors seeing different variants ## Output 1. Test results summary: conversion rates, lift, confidence level 2. Statistical significance assessment (significant / not significant / inconclusive) 3. Segment breakdown: results by device, source, new vs. returning 4. Recommendation: implement winner / extend test / redesign test 5. Next test suggestion based on learnings
How it works
Takes your A/B test data and runs statistical significance analysis (frequentist and Bayesian). Segments results by traffic source, device, and user type to identify whether the winner is universal or segment-specific. Flags common errors: insufficient sample size, testing too many variables, or stopping too early.
Use cases
- —Determine if a test has reached significance
- —Segment results to find which audience drove the win
- —Identify if a test was stopped too early
Example prompts
Has my A/B test reached statistical significance?
Did the winning variant win across all segments?
Was this test stopped too early?
Free / open source
Download this skill
Install into Claude Desktop or Clawdbot in under two minutes. Works with any Claude plan.
View on GitHub →Managed / no setup
Use in Ryze AI
Skip the setup. Ryze AI connects directly to your accounts with this skill ready to go.
