Running web experiments through Amplitude Experiment helps test hypotheses, validate ideas, and drive data-informed product decisions. However, once you have a clearly winning variant, Amplitude recommends moving the winning variant to your production code base rather than keeping the experiment live with 100% traffic allocation to the winning variant.
When the experiment concludes
When you reach a point with your experiment where you want to end and decide a winner:
- Analyze the results and confirm a winner. Ensure the experiment reached statistical significance and that the winning variant aligns with business goals.
- Implement the winner in code. Work with your engineering team to replicate the winning experience in your production code base. For more information, see Benefits to migrating your winning variant.
- Deactivate or archive the experiment in Amplitude. Disable the experiment to remove unnecessary logic and prevent accidental reactivation or analysis confusion.
- Document the outcome. Capture experiment details like the goal, key learning, decision made, and implementation follow up in an internal knowledge base.
Activate a feature flag
If your change requires the ability to rollback or an incremental rollout, use a feature flag. Feature flags enable ongoing control, without the overhead required to support experiment logic and metadata.
Benefits to migrating your winning variant
Moving your winning variant to your production code base provides the following benefits:
- Performance and user experience: If you continue to run web experiments at 100%, you add avoidable clientside overhead to your pages. This increases page load execution time and can negatively impact performance, especially at scale. For more information about how Amplitude optimizes for performance, see Web Experiment Performance.
- Technical debt: Long running experiments can add clutter to dashboards and experiment environments. Leaving them active after a decision causes unnecessary configuration overhead, and increases the risk of user-facing errors.
- Platform cost and impresion volume: Every time a client evaluates an experiment, it counts toward your monthly impression volume in Experiment. When you run a test at 100%, even when it no longer provides learning, the experiment still evaluates on each page load. Over time, this increases your costs and creates budgeting inefficiences.