A Hybrid Extended Cox-frailty and Random Forest Framework for Prostate Cancer Survival: Integrating Clinical Interpretability with Machine Learning Performance
Emmanuel Asare Ayim *
Department of Statistics and Actuarial Science, Takoradi Technical University, Sekondi-Takoradi, Ghana.
Benjamin Odoi
Department of Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana.
Henry Otoo
Department of Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana.
*Author to whom correspondence should be addressed.
Abstract
Prostate cancer risk stratification in low-resource clinical settings requires survival-prediction tools that address violations of the proportional hazards assumption, unobserved patient heterogeneity, and nonlinear covariate interactions that classical regression frameworks cannot represent. This study develops a Hybrid Extended Cox–Frailty and Random Forest (Hybrid Cox-RF) model using a retrospective cohort of 200 patients with prostate cancer from a Ghanaian tertiary hospital (161 deaths, 80.5%, over the full follow-up period; 3 competing other-cause deaths, 1.5%; and 36 censored observations, 18.0%). A sequential pipeline was applied: (1) Aalen–Johansen competing-risks quantification; (2) Schoenfeld residual-based testing of the proportional hazards (PH) assumption; (3) extended Cox modelling with time-varying covariates for four confirmed PH violators; (4) gamma frailty correction; (5) six machine-learning classifiers benchmarked by 10-fold stratified cross-validation at a fixed 24-month mortality horizon (112 of 200 patients, 56.0%, died by 24 months); and (6) construction of the Hybrid Cox-RF model by embedding out-of-bag Random Forest logit risk scores derived from the 24-month classifier into the full-follow-up gamma frailty Cox partial likelihood with sandwich variance correction. Random Forest was the best-performing classifier at the 24-month horizon (AUC = 0.750; sensitivity = 0.782; F1 = 0.810); PSA was the dominant predictor (14.2% mean Gini decrease), followed by BMI (11.8%), Stage T4 (9.6%), age (8.3%), and Gleason score (7.1%). For the full-follow-up survival endpoint, the Hybrid Cox-RF achieved a 10-fold cross-validated C-index of 0.743, compared with 0.671 for the standard Cox model (ΔC = +0.072; a 10.7% relative improvement), an AIC of 1364.21 (ΔAIC = 57.7 versus the next-best Extended Cox Frailty model), a Hosmer–Lemeshow p-value of 0.794, and a calibration slope of 0.97. Because classifier tuning and hybrid-model fitting were not incorporated into a fully nested cross-validation loop, these hybrid-model estimates should be interpreted as an internally cross-validated upper bound pending confirmation through external validation or nested cross-validation. The RF risk score was strongly associated with mortality (HR = 10.91 per 0.5-unit logit increase; 95% CI: 5.47–21.77); most clinical covariate hazard ratios remained stable (within ±15%) after its inclusion, the RF score absorbed the linear age effect, and rural residence remained independently significant (HR = 1.597, p = 0.008), indicating structural barriers to healthcare access beyond clinical severity. Pending external validation and nested cross-validation, the Hybrid Cox-RF framework is proposed as a candidate analytical approach for prostate cancer survival prediction in Ghanaian and comparable sub-Saharan African clinical settings, offering an interpretable artificial-intelligence approach that can support precision oncology and clinical decision-making in low-resource healthcare systems.
Keywords: Prostate cancer survival, cox proportional hazards model, random forest, hybrid survival modelling, machine learning classifiers