PREDICTIVE ANALYTICS FOR CUSTOMER CHURN USING LOGISTIC REGRESSION AND RANDOM FOREST

Background to the Study

Customer churn defined as discontinuation of customer relationship with service provider within defined observation period represents one of most persistent and financially consequential challenges facing subscription-based industries including telecommunications banking insurance and streaming media services. Strategic importance churn management well established marketing and customer relationship management literature which long documented cost acquiring new customer typically substantially exceeds cost retaining existing one making accurate statistically grounded churn prediction high-value analytical capability for any subscription-based business. Proliferation of customer relationship management CRM systems and digital service platforms generated increasingly rich granular customer-level data encompassing contractual details billing history service usage patterns and customer service interaction logs that collectively constitute rich substrate for statistical churn prediction modelling. Within analytical landscape binary logistic regression and random forest emerged as two most widely applied and directly comparable churn prediction methodologies: logistic regression classical statistical technique offering directly interpretable coefficients and odds ratios of considerable value for business stakeholder communication and regulatory transparency and random forest ensemble machine learning technique capable of capturing complex non-linear interactions among predictors without requiring analyst to pre-specify functional form. While numerous prior studies compared logistic regression against random forest for churn prediction substantial portion comparative literature relies on narrow evaluation methodology most commonly single train-test split evaluated via accuracy or limited subset classification metrics without incorporating fuller battery statistical validation techniques increasingly regarded as best practice rigorous applied predictive modelling research. This narrower evaluation approach risks both overstating reliability any single-split performance estimate given absence cross-validation to assess estimate stability and understating practical business relevance model comparison given absence explicit linkage between statistical classification performance and actual monetary costs benefits retention decision-making that any deployed churn model would ultimately inform. This study accordingly undertakes substantially more comprehensive statistical evaluation of logistic regression against random forest for customer churn prediction extending beyond simple accuracy comparison to incorporate five complementary evaluation dimensions: standard classification metrics accuracy precision recall F1-score AUC on held-out test set; stratified k-fold cross-validation to assess stability generalisability each model's discriminatory performance across multiple resampled data partitions; McNemar test providing formal statistical significance testing two models' paired classification agreement methodologically more appropriate comparison than treating each model's test-set performance as arising from independent samples; probabilistic calibration assessment via Brier score evaluating not merely each model's binary classification accuracy but reliability its underlying predicted probabilities property direct importance for any application such as targeted retention campaign budgeting that relies on model's probability estimates rather than binary classifications alone; and cost-sensitive profit curve analysis explicitly incorporating asymmetric business costs retention intervention cost false-positive retention offer extended to customer who would not in fact have churned against benefit successful retention value true-positive customer correctly identified and retained translating abstract classification performance into directly interpretable business-value metric. This comprehensive multi-dimensional evaluation framework directly addresses well-recognised gap between academic churn prediction benchmarking practice which frequently emphasises accuracy or AUC in isolation and fuller statistical and business rigour genuinely required for responsible defensible churn model selection and deployment in operational business context. Value integrated approach extends beyond methodological thoroughness alone: by triangulating findings across standard classification metrics cross-validation stability formal paired significance testing probabilistic calibration and cost-sensitive business-value translation study positioned to reveal genuinely nuanced trade-offs such as finding that two models trade off precision accuracy against recall discrimination that single-metric comparison would entirely obscure providing decision-makers fuller multi-dimensional evidentiary basis genuinely required defensible churn model selection decision.

Statement of the Problem

Despite considerable academic industry attention devoted to customer churn prediction many organisations particularly small medium-sized telecommunications subscription-service providers with limited in-house data science capacity continue to select churn prediction models based on single performance metric most commonly accuracy evaluated on single train-test split without incorporating cross-validation to assess result stability formal paired significance testing to determine whether apparent performance difference between candidate models statistically meaningful rather than attributable to sampling variability calibration assessment to verify reliability predicted probabilities or explicit linkage to actual business costs benefits that would determine model's real-world retention-campaign value. Creates meaningful risk organisations select churn prediction model that while performing adequately on specific metric emphasised is suboptimal or even counterproductive when evaluated against fuller set statistical and business criteria genuinely relevant to deployment: model with marginally higher accuracy may upon closer statistical inspection offer materially lower recall missing larger proportion genuine churners customers retention campaign specifically designed to reach poorly calibrated probability estimates undermining probability-based campaign targeting or budget allocation or lower expected retention profit once asymmetric costs false positives false negatives properly accounted for. There is also comparatively limited Nigerian undergraduate statistics research that applies this full integrated evaluation framework standard classification metrics cross-validation McNemar paired testing calibration analysis cost-sensitive profit curve analysis to logistic regression versus random forest churn prediction comparison within single methodologically coherent research project. Against backdrop study seeks statistically compare logistic regression and random forest for customer churn prediction using comprehensive multi-dimensional evaluation framework thereby providing methodologically rigorous business-relevant guidance for churn prediction model selection.

Aim and Objectives of the Study

Aim is to statistically compare logistic regression and random forest for customer churn prediction using comprehensive evaluation framework incorporating cross-validation statistical significance testing calibration analysis cost-sensitive business-value analysis.

·         Determine overall churn rate and describe contractual billing and service-usage characteristics customer sample.

·         Examine bivariate relationships between candidate predictors contract type technical support internet service payment method and churn status.

·         Identify statistically significant independent predictors churn using binary logistic regression.

·         Comparatively evaluate logistic regression and random forest using standard classification metrics accuracy precision recall F1-score AUC on held-out test set.

·         Assess stability generalisability each model's discriminatory performance using stratified 5-fold cross-validation.

·         Formally test whether two models paired classification performance differs significantly using McNemar test.

·         Evaluate and compare probabilistic calibration each model's predicted churn probabilities using Brier score.

·         Translate each model's classification performance into expected retention-campaign profit estimate using cost-sensitive profit curve analysis.

Research Questions

1.      What is overall churn rate among customers in dataset and what are their key contractual service-usage characteristics?

2.      Is there statistically significant relationship between contract type technical support internet service payment method and churn?

3.      Which factors are statistically significant independent predictors of churn?

4.      How do logistic regression and random forest compare on standard classification metrics?

5.      How stable and generalisable is each model's discriminatory performance across cross-validation folds?

6.      Is there statistically significant difference in two models paired classification performance?

7.      Which model produces better-calibrated churn probability estimates?

8.      Which model yields higher expected retention-campaign profit under cost-sensitive evaluation framework?

Research Hypotheses

·         H01: There is no statistically significant difference in mean tenure between churned and retained customers.

·         H02: There is no statistically significant difference in mean monthly charges between churned and retained customers.

·         H03: There is no statistically significant association between contract type and churn.

·         H04: There is no statistically significant association between technical support subscription and churn.

·         H05: Tenure monthly charges customer service calls contract type technical support online security internet service payment method do not significantly predict churn.

·         H06: There is no statistically significant difference in paired classification performance of logistic regression and random forest.

Significance of the Study

Significant to telecommunications and subscription-service providers seeking methodologically rigorous business-relevant basis for selecting between logistic regression and random forest for churn prediction model deployment particularly organisations seeking move beyond simple accuracy-based model selection toward fuller statistical and business-value evaluation framework. For statisticians and data scientists study demonstrates integrated application cross-validation formal paired significance testing calibration assessment cost-sensitive profit analysis within single coherent churn prediction research framework addressing practically important methodological gap in much existing comparative churn prediction literature. Academically serves as comprehensive reference template for students undertaking predictive analytics customer relationship management analytics or comparative statistical/machine learning research.

Scope of the Study

Restricted to statistical analysis structured cross-sectional dataset 1,500 telecommunications customer records comprising contractual contract type payment method billing monthly charges total charges service-usage internet service technical support online security device protection streaming TV demographic senior citizen status partner dependents and behavioural tenure customer service calls variables. Analysis covers descriptive statistics Pearson correlation independent samples t-tests Chi-square tests one-way ANOVA binary logistic regression and comprehensive logistic regression versus random forest comparison incorporating standard classification metrics stratified 5-fold cross-validation McNemar test Brier score calibration analysis and cost-sensitive profit curve analysis. Does not extend to survival-analysis-based time-to-churn modelling additional machine learning algorithms beyond logistic regression random forest or genuine external validation on independent customer population all recommended as areas further research.

Limitations of the Study

Subject to certain limitations common to applied predictive analytics research. First dataset used while structured to reflect realistic relationships effect sizes documented in prior churn prediction literature is simulated illustrative dataset rather than data obtained directly from specific telecommunications provider proprietary customer database; numerical estimates reported throughout study including cost-sensitive profit curve analysis should therefore be interpreted as illustrative statistical methodology rather than provider-specific empirical findings. Second cost-sensitive profit curve analysis relies on simplified illustrative assumptions regarding retention-offer cost intervention success rate; genuine organisational application would require parameters empirically estimated from specific institution actual retention campaign history rather than assumed. Third study compares only two algorithms logistic regression random forest consistent with title scope; broader comparative machine learning evaluation while valuable outside defined scope. Researchers applying methodology to actual organisational dataset should substitute simulated dataset with genuine appropriately anonymised customer records empirically calibrated cost parameters following identical analytical procedure described in Chapter Three.

Operational Definition of Terms

·         Customer Churn: Binary outcome indicating whether customer discontinued subscription coded 1 or remained active coded 0 within observation period.

·         Cross-Validation: Model evaluation technique partitions data into multiple folds iteratively training on subset validating on remainder to obtain more robust stable estimate model generalisation performance than single train-test split provides.

·         McNemar Test: Statistical test comparing paired classification performance two models applied to same set observations based on discordant classification pairs.

·         Brier Score: Proper scoring rule quantifying mean squared difference between predicted probabilities and actual binary outcomes with lower values indicating better probabilistic calibration.

·         Cost-Sensitive Analysis: Evaluation approach explicitly incorporates differing real-world costs benefits associated with different types classification outcomes true positives false positives true negatives false negatives into model performance assessment.

·         Profit Curve: Visualisation tabulation classification model expected business profit or loss across different classification thresholds or model choices incorporating specific cost-benefit structure deployment context.

Short Conclusion

Results showed overall churn rate 31.67% 475 of 1,500. Contract type χ2=108.03 p<0.001 technical support χ2=28.43 p<0.001 strongly associated with churn and churned customers had significantly shorter tenure 28.45 vs 35.40 months t=-8.325 p<0.001 significantly higher monthly charges 68.12 vs 63.70 t=3.916 p<0.001 significantly more service calls 1.91 vs 1.50 t=5.634 p<0.001. Logistic regression identified tenure monthly charges service calls contract type technical support online security senior citizen as significant predictors McFadden pseudo R2=0.166. On held-out test set Random Forest marginally higher accuracy 71.33% vs 70.89% precision 55.30% vs 53.37% alongside modestly better Brier 0.1903 vs 0.1967 while Logistic Regression substantially higher recall 66.43% vs 51.05% higher AUC 75.43% vs 73.62% higher 5-fold CV mean AUC 76.30% vs 75.31% lower CV variance. McNemar test found no significant difference paired classification error patterns χ2=0.016 p=0.901 indicating despite differing profiles neither significantly outperforms other in overall paired classification agreement. Cost-sensitive profit curve analysis incorporating assumed retention-offer economics projected substantially higher expected retention profit under Logistic Regression ₦23,057.41 than Random Forest ₦17,885.80 driven primarily by Logistic Regression superior recall greater capture at-risk customers eligible for retention intervention. Concludes model selection should be explicitly grounded in specific business decision context rather than single default metric with Logistic superior recall discrimination and under assumed cost structure superior projected retention profit making stronger candidate proactive retention campaign targeting while Random Forest superior precision calibration may better suit applications prioritising minimisation unnecessary retention-offer costs.

10 SEO-Friendly FAQs

1. What was overall churn rate?

31.67% 475 of 1,500 telecommunications customers; descriptive Pearson t-tests Chi-square ANOVA employed to characterise contractual billing service-usage demographic behavioural variables.

2. Which variables associated with churn bivariately?

Contract type χ2=108.03 p<0.001 and technical support subscription χ2=28.43 p<0.001 strongly associated; internet service payment method also examined; churned had shorter tenure 28.45 vs 35.40 months t=-8.325 p<0.001 higher monthly charges 68.12 vs 63.70 t=3.916 more service calls 1.91 vs 1.50 t=5.634.

3. Which predictors significant in logistic regression?

Tenure monthly charges customer service calls contract type technical support online security senior citizen status statistically significant predictors McFadden pseudo R2=0.166 indicating modest explanatory power but significant effects.

4. How did Logistic Regression vs Random Forest compare on accuracy?

Random Forest marginally higher accuracy 71.33% vs 70.89% and precision 55.30% vs 53.37% on held-out test set; difference small.

5. Which model had better recall and AUC?

Logistic Regression substantially higher recall 66.43% vs 51.05% higher AUC 75.43% vs 73.62% and higher 5-fold stratified cross-validated mean AUC 76.30% vs 75.31% with lower variance indicating more stable discriminatory performance.

6. What about calibration?

Random Forest modestly better lower Brier calibration score 0.1903 vs 0.1967; Brier proper scoring rule mean squared difference predicted probabilities actual binary outcomes lower better; both models evaluated via reliability curve ablation out-of-fold predictions.

7. Did McNemar test show significant difference?

No χ2=0.016 p=0.901 found no statistically significant difference in two models paired classification error patterns indicating despite differing performance profiles across individual metrics neither significantly outperforms other in overall paired agreement ground truth; McNemar based discordant pairs.

8. Which model yields higher expected retention profit?

Logistic Regression projected substantially higher expected retention profit ₦23,057.41 vs Random Forest ₦17,885.80 under cost-sensitive profit curve analysis incorporating assumed retention-offer economics driven primarily by superior recall greater capture at-risk customers eligible for retention intervention.

9. Why use profit curve not accuracy alone?

Profit curve translates classification performance into expected business profit incorporating asymmetric costs false-positive retention offer cost vs true-positive retention benefit; threshold maximising expected profit chosen deployment threshold rather than default 0.5; reveals trade-offs accuracy vs recall business-value.

10. Where to find similar predictive analytics project topics?

Explore customer churn predictive analytics topics on ScholarNestHub data science collection and reproducible workflows customer churn calibrated probability 5-fold CV Brier profit curve.