Membership Org Predicts Churn with ML Model to Retain Community

An organization with an 18,000 membership provides ongoing education, tools and publications to advance their field and drive best practices among practitioners. As a member-driven organization, providing relevant services to its members is key to member retention.  

They engaged SPR to develop a machine learning model to determine what members were likely to leave the organization, or “churn.” Membership churn models are a common application of ML, and the client’s technology team looking to determine membership churn were familiar with data and ML, but they hadn’t put models into practice before. By predicting members who were likely to churn through ML, the client could reach out to key members at risk of leaving the organization and offer incentives for them to stay. 

While the client was already performing ML with an automated ML tool, they were looking for a trusted advisor to review the results they were receiving from the tool and perform an independent machine learning effort to validate the results and suggest additional techniques to improve their model.  

While the client had significant data for 2016 through 2019, the data from 2020 presented a unique challenge: the organization provided free membership extensions in 2020 due to the COVID-19 pandemic. This created anomalies in the data and a large class imbalance; with few examples of membership churn to draw on, determining the pattern of membership churn became a challenge. This is known as a class imbalance issue.  

Working with the client’s Business Analyst, SPR collected and analyzed the data to determine what was and wasn’t helpful for building the ML model. The SPR team also helped the client team understand the data, and curate, prepare and cleanse the data.  

To create a model that would accurately predict churn, SPR had to address the class imbalance issue, and did so by weighting the data of members who churned more heavily.  

SPR created multiple models to determine what factors might cause a member to churn. Through feature engineering our team was able to extract useful information from data that the client thought was relevant to customer churn. These factors were transformed into a form that was able to be fed into a machine learning model. After the model was created, SPR was able to obtain the importance each factor had on the model. This gave the client an idea of which factors led to customer churn and which factors had no bearing on customers leaving. This ultimately led the organization to adjust their thinking on the important aspects of retaining membership. 

The organization was able to successfully put the model into production and estimate the likeliness of a member to churn with an accuracy over 85%. Through collaborating closely with the SPR team, the client team was also able to delve more deeply into data science and re-engage their at-risk members.  

Technologies Used: Python Data Science Stack, Jupyter Notebook, Pandas, NumPy Scikit-Learn, Imbalanced-learn, Matplotlib, Seaborn