The evolution of AI from passive prediction to agentic decision-making, rooted in RL and control and now accelerated by LLMs, surfaces new generalization challenges: deployed agents must cope with never-before-seen conditions, and execute tasks that themselves may not have been seen, all while obeying safety requirements. In this talk I will use linear-quadratic control as a theoretical testbed for studying two important aspects of generalization in AI agents. First, I will address generalization across initial states, showing that the extent to which it succeeds under policy gradient training depends on the degree of exploration induced by the system when commencing from initial states seen in training. Second, I will turn to generalization across tasks, establishing that it is more difficult to achieve with safety requirements (formalized as $H_infty$-robustness) than without, regardless of how well safety requirements are met on tasks seen in training. Experiments demonstrate that the conclusions of the analyses extend to non-linear control with a neural network agent, and to customer relationship management with an LLM agent. Our findings suggest that common efforts to enhance generalization in AI agents may be insufficient, and that learning new representations is a promising alternative.