Skip to content
alyn
phuryn

Ab Test Analysis skill

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

$npx skills add phuryn/pm-skills --skill ab-test-analysis
GitHub stars
27,849
Pack installs
245k
Alyn Score
74/99
Updated
2 days ago
License
MIT
Product management Testing Data analysis

Install Ab Test Analysis

Pick your agent. Every command is ready to paste.

$npx skills add phuryn/pm-skills --skill ab-test-analysis

The open skills CLI detects your agents (Claude Code, Cursor, Codex and 70+ more) and installs to the right folder. Add -g to install globally.

01

Install it

Run the command above, or upload the folder in Claude.

02

Ask naturally

Describe the task. Your agent loads ab-test-analysis when the request matches its description.

03

Name it if needed

If it does not trigger, say “use the ab-test-analysis skill”. In Codex, type $ab-test-analysis.

The skill itself

SKILL.md

GitHub

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions

  1. Understand the experiment:

    • What was the hypothesis?
    • What was changed (the variant)?
    • What is the primary metric? Any guardrail metrics?
    • How long did the test run?
    • What is the traffic split?

  2. Validate the test setup:

    • Sample size: Is the sample large enough for the expected effect size?
      • Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
      • Flag if the test is underpowered (<80% power)
    • Duration: Did the test run for at least 1-2 full business cycles?
    • Randomization: Any evidence of sample ratio mismatch (SRM)?
    • Novelty/primacy effects: Was there enough time to wash out initial behavior changes?

  3. Calculate statistical significance:

    • Conversion rate for control and variant
    • Relative lift: (variant - control) / control × 100
    • p-value: Using a two-tailed z-test or chi-squared test
    • Confidence interval: 95% CI for the difference
    • Statistical significance: Is p < 0.05?
    • Practical significance: Is the lift meaningful for the business?

    If the user provides raw data, generate and run a Python script to calculate these.

  4. Check guardrail metrics:

    • Did any guardrail metrics (revenue, engagement, page load time) degrade?
    • A winning primary metric with degraded guardrails may not be a true win

  5. Interpret results:

    Outcome Recommendation
    Significant positive lift, no guardrail issues Ship it — roll out to 100%
    Significant positive lift, guardrail concerns Investigate — understand trade-offs before shipping
    Not significant, positive trend Extend the test — need more data or larger effect
    Not significant, flat Stop the test — no meaningful difference detected
    Significant negative lift Don't ship — revert to control, analyze why

  6. Provide the analysis summary:

    ## A/B Test Results: [Test Name]

**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]

| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |

**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

Further Reading

Tips for Ab Test Analysis

  • • Start a fresh session after installing so your agent picks up the new skill.
  • • It is part of a pack with 40 skills. Install only the ones you need with --skill to keep triggers precise.
  • • If two skills overlap, mention the one you want by name.

Questions about Ab Test Analysis

What does the Ab Test Analysis skill do?

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

How do I install Ab Test Analysis?

Run "npx skills add phuryn/pm-skills --skill ab-test-analysis" in your project. The skills CLI installs it for Claude Code, Cursor, Codex and 70+ other agents. In Claude on the web, upload the skill folder as a ZIP under Customize > Skills.

Does Ab Test Analysis work in Cursor and Codex?

Yes. Ab Test Analysis uses the open SKILL.md format, so it works in Claude Code, Claude, Cursor, OpenAI Codex, Gemini CLI, GitHub Copilot and other agents that support agent skills.

Is Ab Test Analysis free?

Ab Test Analysis is open source on GitHub under the MIT license. It is free to install; check the license before commercial use.

Who made Ab Test Analysis?

Ab Test Analysis is published by phuryn in the phuryn/pm-skills repository, together with 39 other skills.

Similar skills. From other creators.