Abstract: Data-driven AI weather models are fundamentally different from the true climate system, and from the numerical models that have traditionally been used to predict it. Here, I ask how ideas from dynamical systems, statistics, and climate science can help us interpret when and why these models can be trusted, with a focus on probabilistic prediction. Using the two-scale Lorenz 96 system, I show how coarse-graining a deterministic, Markovian system can naturally lead to stochastic and potentially non-Markovian dynamics, providing one interpretation of spread in generative weather models. I then consider a second source of uncertainty: finite sampling of the climate system, which can lead to substantial differences in estimated distributions even over multi-decadal periods, particularly for extremes. Together, these perspectives raise questions about whether AI models can generate events that are consistent with the underlying climate system but absent from the training data, and what additional information may be needed when they cannot.