Renewable-energy systems increasingly rely on AI-based forecasting to balance variable generation, storage, and demand. Yet evidence remains limited on whether more complex models consistently improve operational decisions under real-world distribution shifts. We conducted an empirical study across 240 renewable-energy sites, comprising 120 solar and 120 wind installations, using three years of hourly generation and weather data. We compared gradient-boosted trees and transformer-based forecasting models against a seasonal persistence baseline. Models were evaluated on held-out periods containing extreme weather, sensor outages, and demand anomalies; primary outcomes were forecast error and simulated curtailment. Contrary to expectations, advanced models performed worse under distribution shift. The transformer increased mean absolute forecast error by 8.7% relative to seasonal persistence (14.9% versus 13.7%) and raised simulated curtailment by 5.2%. Gradient-boosted trees produced no significant improvement in forecast error (13.6%) and reduced curtailment by only 0.4%. At 63% of sites, at least one advanced model underperformed the baseline. These findings challenge the assumption that increasing model complexity necessarily supports renewable-energy integration. Our study contributes a large, site-level evaluation framework for stress-testing energy forecasts and shows that robustness, monitoring, and fallback baselines may matter more than marginal gains on standard test sets. Future renewable-energy software should prioritize shift-aware evaluation and operational resilience before deploying complex forecasting systems. Program booklet (40 words): Across 240 solar and wind sites, advanced forecasting models unexpectedly underperformed a seasonal baseline during real-world disruptions. Transformers increased forecast error by 8.7% and curtailment by 5.2%, highlighting the need for robust evaluation, monitoring, and fallback strategies in renewable-energy s