RainCheck 研究揭示孟加拉国降雨预测中的数据泄露问题
RainCheck: Harnessing Historical Weather Data with Machine Learning for Rainfall Forecasting in Bangladesh
这篇论文用孟加拉国 66 年数据证明：很多降雨预测模型的"高分"其实是数据泄露，气候学基线就能打平大部分机器学习模型，做时序预测的都该看看。
研究基于孟加拉国气象部门 35 个站点 1948-2014 年共 618,650 个站点日数据，比较随机、时间和时空三种评估方式。结果显示仅用站点-月份气候学基线就能以 88.4% 准确率、0.950 AUC 分类高降雨月份，随机切分下日历特征的随机森林 AUC 达 0.889，但在未来年份降至 0.835、低于气候学基线的 0.859。加入滞后降雨特征后模型获得真实但有限的技能，对气候学的 Brier skill score 为 0.074，且技能在超过两天预报时效后消失。团队发布了防泄漏评估协议和代码。
RainCheck: Harnessing Historical Weather Data with Machine Learning for Rainfall Forecasting in Bangladesh
Machine-learning rainfall forecasts are often evaluated by shuffling station records into random training and test sets. For rainfall, this lets a model see what happened at neighbouring stations on the very day it is asked to forecast, and it hides the fact that a seasonal lookup table already explains most of the variance. We quantify both effects on 618,650 station-days from 35 Bangladesh Meteorological Department stations (1948-2014). We compare random, temporal (train <=1999, test 2005-2014) and spatio-temporal (unseen stations and unseen years) evaluation, calendar-only versus antecedent-rainfall features, and six learners against climatology, persistence and SARIMA. Three findings stand out. First, a station-month climatology alone classifies high-rainfall months with 88.4% accuracy (AUC 0.950), and no learner improved on it by more than 0.8 points under temporal evaluation. Second, a calendar-only random forest reached a next-day wet-day AUC of 0.889 under a random split, higher than any model with real predictors under temporal evaluation, yet fell to 0.835, below climatology (0.859), on future years. Third, models with lagged rainfall add genuine but modest skill for next-day forecasts: a Brier skill score of 0.074 (95% CI 0.064-0.088) over climatology, rising to 0.091 when station and year inputs are removed and to 0.131 when other stations' same-day observations are added. A GRU sequence model matches but does not exceed gradient boosting, and skill disappears beyond a two-day lead. For monthly totals, no learner beat climatology significantly. We release a leakage-aware protocol and code so that future station-based studies report skill against climatology on held-out years.
