Appearance
Experiments & A/B Testing
Experiments let you run controlled tests—for example, different prompts, models, or features—measure outcomes, and iterate safely. The system supports a full experiment lifecycle, variants, results analysis, and event tracking for attribution.
Experiment lifecycle
Experiments progress through well-defined states:
| State | Meaning |
|---|---|
| draft | Being configured; not yet exposing traffic. |
| running | Actively assigning users or sessions to variants and collecting data. |
| paused | Temporarily stopped; no new assignments until resumed. |
| completed | Ended; results are retained for analysis. |
Use draft for design, running for live data, paused for incidents or maintenance, and completed when you have a decision or the test has run its course.
Experiment IDs (user-provided)
Experiment identifiers are supplied by you, not auto-generated by the platform. Choose stable, meaningful IDs (for example, prompt_v2_jan_2026) so logs, dashboards, and external systems stay aligned.
If you attempt to create an experiment whose ID already exists, the API returns HTTP 400 (bad request) indicating a duplicate—you must pick a new ID or update the existing experiment through the appropriate API.
Variants
Each experiment defines variants (for example, control vs. treatment). Assignment logic routes subjects—often users or sessions—to a variant so metrics can be compared fairly. Document your variant definitions alongside the experiment ID for reproducibility.
Results analysis
After data collection, results analysis compares metrics across variants: conversion, latency, quality scores, or custom events you track. Tie analysis back to the same experiment ID and time window for clear conclusions.
Event tracking
Event tracking records outcomes (exposures, clicks, completions, errors) with enough context to attribute them to an experiment and variant without breaking privacy policies. Use consistent event names and properties across variants.
Best practices
- Define success metrics before setting the experiment to running.
- Prefer user-provided IDs that encode team or initiative for easier operations.
- Handle 400 responses on duplicate IDs by renaming or reconciling with existing experiments.