Reliable Uncertainty Quantification for Time Series Forecasting: A Unified Framework for Benchmarking

Reliable uncertainty quantification is essential for critical decisions in time series forecasting, yet many machine learning models focus solely on point estimates. This paper presents a systematic evaluation of regression models, including Linear Regression, Random Forest, and three state-of-the-art boosting models: eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), and Categorical Boosting (CatBoost), combined with the Conformal Prediction (CP) framework to generate statistically valid prediction intervals. A unified and reproducible evaluation pipeline is proposed, integrating time-aware cross-validation, lag-based feature engineering, and calibration-based split-conformal prediction. The framework is evaluated on four heterogeneous real-world datasets from the energy, transportation, weather, and retail domains. Results show that empirical coverage frequently falls below the theoretical target due to violations of the exchangeability assumption, while boosting-based models achieve up to 30\% lower pinball loss and more stable prediction intervals.