Skip to main content

Rollouts & Kill Switches

Learn how to safely roll out features and use kill switches to quickly disable features in emergencies.

Feature Rollouts​

A rollout is the process of gradually enabling a feature for an increasing percentage of users.

Rollout Strategy​

Phase 1: Internal Testing (1-5%)​

Start with a small percentage to catch issues early:

  1. Enable for 1% of users
  2. Monitor for 24-48 hours
  3. Check metrics, error rates, and user feedback
  4. Increase to 5% if no issues

Phase 2: Early Adopters (5-25%)​

Expand to early adopters:

  1. Increase to 5% of users
  2. Monitor for another 24-48 hours
  3. Increase to 10%, then 25%
  4. Watch for edge cases and unusual patterns

Phase 3: Majority (25-100%)​

Roll out to the majority of users:

  1. Increase to 50% of users
  2. Monitor closely for issues
  3. If stable, increase to 100%
  4. Consider keeping at 95% as a safety margin

Rollout Best Practices​

  • Start Small: Always begin with 1-5%
  • Monitor Continuously: Watch metrics throughout the rollout
  • Increase Gradually: Don't jump from 10% to 100%
  • Have Rollback Plan: Know how to disable quickly
  • Document Decisions: Record why you're rolling out and what you're watching

Kill Switches​

Kill switches are feature flags used to quickly disable features in emergencies without code deployment.

Creating Kill Switches​

  1. Create a feature flag for critical features
  2. Name it clearly (e.g., kill-switch-payment-gateway)
  3. Set up monitoring and alerts
  4. Document the rollback procedure

Using Kill Switches​

// Critical payment feature
const paymentEnabled = await toggly.isFeatureEnabled('new-payment-gateway');

if (!paymentEnabled) {
  // Fall back to legacy payment system
  return processLegacyPayment();
}

// Use new payment gateway
return processNewPayment();

Kill Switch Best Practices​

  1. Create for Critical Features: Use kill switches for features that could cause major issues
  2. Test Regularly: Periodically test that kill switches work
  3. Monitor Alerts: Set up alerts for kill switch activations
  4. Document Procedures: Document when and how to use kill switches
  5. Train Team: Ensure team members know how to use kill switches

Emergency Procedures​

When to Use a Kill Switch​

Use kill switches when:

  • High Error Rates: Error rates spike unexpectedly
  • Performance Degradation: Response times increase significantly
  • Business Metrics Drop: Conversion rates or revenue drop
  • User Complaints: Significant increase in support tickets
  • Security Issues: Potential security vulnerabilities detected

How to Activate a Kill Switch​

  1. Navigate to the feature flag in the dashboard
  2. Click Disable (or set percentage to 0%)
  3. Verify the feature is disabled
  4. Monitor metrics to confirm the issue is resolved
  5. Investigate the root cause
  6. Fix the issue before re-enabling

Post-Incident​

After using a kill switch:

  1. Document the Incident: Record what happened and why
  2. Root Cause Analysis: Investigate the root cause
  3. Fix the Issue: Resolve the underlying problem
  4. Test Thoroughly: Test the fix before re-enabling
  5. Gradual Re-enablement: Re-enable gradually, not all at once

Monitoring During Rollouts​

Key Metrics to Watch​

  • Error Rates: Percentage of failed requests
  • Performance: Response times and latency
  • Business Metrics: Conversion rates, revenue
  • User Feedback: Support tickets, user complaints
  • Feature Usage: How often the feature is used

Setting Up Alerts​

Configure alerts for:

  • Error Rate Thresholds: Alert if error rate exceeds X%
  • Performance Degradation: Alert if latency increases by X%
  • Business Metric Drops: Alert if conversions drop by X%
  • Anomaly Detection: Alert on unusual patterns

Rollout Examples​

Example 1: New Checkout Flow​

Day 1: Enable for 1% of users
Day 2: Monitor metrics, no issues → increase to 5%
Day 4: Monitor metrics, no issues → increase to 10%
Day 6: Monitor metrics, no issues → increase to 25%
Day 8: Monitor metrics, no issues → increase to 50%
Day 10: Monitor metrics, no issues → increase to 100%

Example 2: Premium Feature​

Week 1: Enable for premium users only
Week 2: Monitor metrics and feedback
Week 3: Expand to premium + enterprise users
Week 4: Monitor metrics
Week 5: Roll out to all users if successful

Best Practices​

1. Always Have a Rollback Plan​

Know how to disable features quickly before starting a rollout.

2. Monitor Continuously​

Watch metrics throughout the rollout, not just at the end.

3. Start Small​

Always begin with a small percentage (1-5%) to catch issues early.

4. Increase Gradually​

Don't jump from 10% to 100%. Increase in stages.

5. Document Everything​

Record rollout decisions, metrics, and any issues encountered.

Next Steps​