The subscription business was losing 6% of its customers every month — a 72% annual churn rate. For every 100 customers acquired in January, only 28 were still paying by December. Three months after we shipped a churn model, monthly churn was down to 4.2% and retention spend was finally landing on the right people.
TL;DR
- Client: A subscription business with 180,000 customers bleeding 6% per month.
- Problem: Retention was one generic discount blasted at everyone — annoying loyal customers, missing the at-risk ones.
- Result: Monthly churn 6.0% → 4.2% (≈30% reduction), ~$340K in annual recurring revenue retained, and 3.5× higher retention among model-targeted customers vs. untargeted campaigns.
The Problem
The marketing team was running retention campaigns, but blasting everyone with the same discount. High-value loyal customers got annoyed. At-risk customers got the same generic email as everyone else. The campaigns were expensive and barely moved the needle. They needed to know which customers were about to leave and why — before it happened.
What Was at Stake
At a 72% annual churn rate, the company was refilling a leaking bucket: nearly three-quarters of every cohort had to be re-acquired within a year just to stand still. Acquisition costs were climbing while the discount-everyone strategy was actively eroding margin on the customers who were never going to leave. Left alone, the unit economics didn’t work.
The Approach
The data. Three years of history for 180,000 customers: subscription events, transaction history, engagement signals (logins, feature usage, tickets, email opens), and B2B demographics. Target: did the customer cancel within the next 30 days? Binary classification.
Feature engineering did the heavy lifting (more impact than model choice):
- RFM — recency, frequency, monetary per customer.
- Behavioral trends — login velocity (7-day vs 30-day), feature-adoption drop, support escalation, payment friction. A static snapshot misses the customer who logged in 20× last month and 3× this month.
- Interaction features — “high spend + declining engagement” (the silent churner), “recent ticket + low satisfaction,” “long tenure + first-ever downgrade.” ~60 features built; ~20 mattered.
Model selection. XGBoost won on both accuracy and speed (AUC-ROC 0.87, Precision@10% 0.68) over logistic regression, random forest, and an MLP that was 15× slower for slightly worse results. For engineered tabular data, gradient boosting still beats deep learning.
Why Precision@10% over AUC: marketing doesn’t target everyone — it targets the top 10% most at-risk. At 68% precision, 7 of 10 customers who got a retention offer were genuinely at risk — enough to justify the spend.
Interpretation with SHAP. A score with no reason is useless to the business. SHAP surfaced the top driver per customer — an 80% login drop, three unresolved tickets in a week, a downgrade after 18 months of premium — so retention could personalize: re-engagement email, priority support escalation, or an account-manager call.
Delivery. A weekly Python batch scored every customer into risk tiers and pushed to a Tableau dashboard the marketing team used every Monday: risk-tier action list, per-customer churn drivers, cohort trends, and a campaign tracker that compared targeted customers against a control group to close the loop.
The Results
After three months of model-driven retention:
- Monthly churn: 6.0% → 4.2% (≈30% reduction)
- Revenue retained: ~$340K in annual recurring revenue
- Campaign efficiency: 3.5× higher retention among model-targeted vs. untargeted customers
- False positives: only 32% of “at-risk” flags were false alarms — acceptable for the campaign cost
What I’d Do Differently
We wasted a sprint on a Cox survival model to predict when a customer would churn — the predictions (“sometime in the next 2–8 weeks”) were too noisy to time a campaign around. The simple binary 30-day model was both simpler and more useful. Next time I’d resist the urge to over-model and ship the binary classifier first, then only add sophistication if the business proves it needs it. The neural network was the same lesson: complexity that added latency, not accuracy.
The Stack
| Component | Tool | Why |
|---|---|---|
| Feature Engineering | Python, Pandas | Fast iteration on 60+ features |
| Model Training | XGBoost, scikit-learn | Best accuracy on tabular data, fast training |
| Model Interpretation | SHAP | Per-customer churn explanations |
| Dashboard | Tableau | Marketing team’s preferred BI tool |
| Batch Pipeline | Python + cron | Weekly scoring, export to Tableau |
Simba Hu helps companies make better decisions with data and AI — from strategy to implementation. Based in Tokyo, serving clients globally. Book a strategy call or visit simbahu.com.