Decision Making and Uncertainty

We study Merton’s expected utility maximization problem in an incomplete market, characterized by a factor process in addition to the stock price process, where all the model primitives are unknown. We take the reinforcement learning (RL) approach to learn optimal portfolio policies directly by exploring the unknown market, without attempting to estimate the model parameters. Based on the entropy-regularization framework for general continuous-time RL formulated in Wang et al. (2020), we propose a recursive weighting scheme on exploration that endogenously discounts the current exploration reward by past accumulative amount of exploration. Such a recursive regularization restores the optimality of Gaussian exploration.  However, contrary to the existing results, the optimal Gaussian policy  turns out to be biased in general, due to the interwinding  needs for hedging and  for exploration.  We present an asymptotic analysis of the resulting errors to show how the level of exploration affects the learned policies. Furthermore, we establish a policy improvement theorem and design several RL algorithms to learn  Merton’s optimal strategies. Finally, we carry out both simulation and empirical studies with a  stochastic volatility environment to demonstrate the efficiency and robustness of the RL algorithms in comparison  to the conventional  plug-in method.

Over the past 30 years, nearly all the gains in the U.S. stock market have been earned overnight, while average intraday returns have been negative or flat. We find that a large part of this effect can be explained by differences between intraday and overnight news. Our analysis uses a collection of over a million news articles. We apply a novel technique for supervised topic analysis that selects news topics based on their ability to explain contemporaneous market returns. We find that differences in the mix of news topics and differences in the responses to news topics both contribute to the difference in intraday and overnight returns. In out-of-sample tests, our approach forecasts which stocks will do particularly well overnight and particularly poorly intraday. We contrast the effect of news with other mechanisms proposed in the literature to explain overnight returns. This is joint work with Kriste Krstovski, Paul Laliberte, and Harry Mamaysky.

We study binary opinion formation in a large population where individuals are influenced by the opinions of other individuals. The population is characterised by the existence of (i) communities where individuals share some similar features, (ii) opinion leaders that may trigger unpredictable opinion shifts in the short term (iii) some degree of incomplete information in the observation of the individual or public opinion processes. In this setting, we study three different approximate mechanisms: common sampling approximation, independent sampling approximation, and, what will be our main focus in this paper, McKean-Vlasov (or mean-field) approximation. We show that all three approximations perform well in terms of different metrics that we introduce for measuring population level and individual level errors. In the presence of a common noise represented by the major influencers opinions processes, and despite the absence of idiosyncratic noises, we derive a propagation of chaos type result. For the particular case of a linear model and particular specifications of the major influencers opinion dynamics, we provide additional analysis, including long term behavior and fluctuations of the public opinion.The theoretical results are complemented by some concrete examples and numerical analysis, illustrating the formation of echo-chambers, the propagation of chaos, and phenomena such as snowball effect and social inertia.

We consider a problem of stochastic optimal control with separable drift uncertainty in strong formulation on a finite horizon. The drift coefficient of the state   is multiplicatively influenced by an unknown random variable, while admissible controls are required to be adapted to the observation filtration. Choosing a control actively influences the state and information acquisition simultaneously and comes with a learning effect. The problem, initially non-Markovian, is embedded into a higher-dimensional Markovian, full information control problem with control-dependent filtration and noise. To that problem, we apply the stochastic Perron method to characterize the value function as the unique viscosity solution to the HJB equation, explicitly construct  ε-optimal controls and show that the values of strong and weak formulations agree. Numerical illustrations show a significant difference between the adaptive control and the certainty equivalence control.

The optimal transport problem can admit many solutions, for instance when the cost function is the Euclidean distance on R^d (Monge’s problem). On the other hand, entropically regularized optimal transport (EOT) always has a unique optimizer. It is a longstanding open problem whether EOT selects some particular optimal transport coupling in the vanishing regularization limit, and what that coupling would look like. We provide a surprising answer for Monge’s problem in dimension d > 1. (Joint work with Chenyang Zhong.)

We study the uncertain transition to a carbon-neutral economy. The requisite technological innovation is made more probable through research and development (R&D). We explore multiple channels of economic and geoscientific uncertainties that impact this transition, and we show how to assess the relative importance of their varied contributions. We represent the social benefit of R&D and cost of global warming as expected discounted values of social payoffs using a probability measure adjusted for concerns about model misspecification and prior ambiguity. Our quantitative results show the value of R&D investment even when the timing of its  technological success is highly uncertain.

Choquet capacities and integrals are useful concepts for modeling decision under uncertainty, pioneered by Schmeidler. We study a class of functionals, called Choquet quantiles, which are Choquet integrals with respect to binary capacities. Choquet quantiles are a generalization of probabilistic quantiles, also known as Value-at-Risk in finance. Choquet quantiles enjoy many nice properties of probabilistic quantiles, in the contexts of axiomatic representation, optimization formulas, and risk sharing. In particular, Choquet quantiles are characterized by only one axiom, called ordinality, and the inf-convolution of Choquet quantiles is again a Choquet quantile. Moreover, we introduce the Choquet Expected Shortfall, giving rise to a new class of coherent risk measures under uncertainty.

Agents in complex economic networks face intrinsic uncertainty regarding global network structure. As real networks are large and complex, even small network uncertainties can lead to huge uncertainties about the market values and risks (i.e., high parameter sensitivity) of firms and organizations in the face of network cascades. This raises the question of how organizations, regulators, or investors can utilize network models to assess the risks at the organization level despite imperfect information. We derive a solution to this challenge and a new unifying perspective. We apply perturbation theory based on conditioning to quantify the sensitivity of node values to uncertainty in network parameters in the presence of nonlinear cascade effects. Our main result is a new efficient algorithm to bound uncertainty in node values in models with nonlinear cascade effects given uncertainty ranges in parameters. We illustrate the application of this algorithm on a real world network. Our applied case study demonstrates the efficiency of our methods in uncovering valuable information into the risks faced by numerous nodes in the network, even in the presence of significant sensitivity challenges posed by network cascade models.” (Joint with Ariah Klages-Mundt)

The Bass local volatility model introduced by Backhoff-Veraguas–Beiglböck–Huesmann–Källblad is a Markov model perfectly calibrated to vanilla options at finitely many maturities, that approximates the Dupire local volatility model. Recently Conze and Henry-Labordère showed that its calibration can be achieved by solving a fixed-point equation. In this talk I will present our results of existence and uniqueness of the solution to this equation, and of convergence of the fixed-point iteration scheme. This talk is based on joint work with A. Marini and G. Pammer.

We first consider Markov decision processes where the state of the chain is only given at chosen observation times and of a cost. Optimal strategies involve the optimisation of observation times as well as the subsequent action values. We derive a quasi-variational inequality for the value function, show its well-posedness, and propose an efficient penalty scheme for the numerical solution. The model and numerical scheme are illustrated by an example from medical testing and treatment. We then consider an extension in which agents can exercise control actions that affect their speed of access to information. We will study single as well as multi-agent settings and their mean-field limit. The agents can dynamically decide to receive observations with less delay by paying higher observation costs. Agents seek to exploit their active information gathering by making further decisions to influence their state dynamics to maximize rewards. In the mean field equilibrium, each generic agent solves individually a partially observed Markov decision problem in which the way partial observations are obtained is itself also subject of dynamic control actions by the agent. We prove that with sufficient entropy regularisation, a fixed point iteration converges to the unique MFG equilibrium and yields an approximate Nash equilibrium for a large but finite population size. We illustrate our MFG by an example from epidemiology, where medical testing results at different speeds and costs can be chosen by the agents. Joint work with Dirk Becherer and Jonathan Tam.

In this talk we will discuss discounted infinite horizon mean field games in a fully probabilistic setting. We start by presenting well-posedness results for the game, which extend well-established analogous results in the finite horizon case. The key ingredient of existence is to construct local versions of the stable topologies usually considered in the finite horizon case. We will then discuss various long time asymptotic results including qualitative and quantitative convergence of finite horizon mean field games to infinite horizon games and to stationary mean field games. The talk is based on a joint work with Rene Carmona and Kaiwen Zhang.

Reinforcement Learning (RL) is a powerful framework for solving sequential decision-making problems,  Fast convergence and sample efficiency are critical for many applied RL problems, such as financial trading and healthcare treatment recommendations, where acquiring new samples is costly or the chance of exploring new actions in the system is limited. In such cases, the cost of making incorrect decisions can be prohibitively high. In this talk, we present two new policy learning methods for a class of discounted linear-quadratic control (LQC) problems over an infinite time horizon with entropy regularization, and show their linear and super-linear convergence in finding optimal policies. We also discuss how to apply transfer learning techniques in such an RL setting. Based on joint work with Xinyu Li of UC Berkeley and Renyuan Xu of USC.