Tara Javidi

The physical world generates far more information than our sensors can and do capture today — early signs of anomalies go undetected not because they are undetectable, but because sensors lack the intelligence to pay attention to them. Physical intelligence has the promise to close that gap, transforming our relationship with the built and natural environment from reactive to predictive — with implications ranging from averting industrial disasters and wildfires to safeguarding aging infrastructure and enabling early warning of public health crises.

In this lecture, we introduce and discuss physical intelligence for monitoring and awareness at scale. We start by framing the problem: how can we utilize advances in sensing and connectivity to understand and effectively monitor the physical world at scale? We argue that physical intelligence at scale is fundamentally a stochastic decision problem with partial observations. More specifically, this provides a general formulation that accounts for both the physics of sensing (noisy and imperfect) as well as the spatio-temporal structure of the environment. In this lecture, I show how this formulation generalizes Shannon’s canonical problem of joint source—channel coding with feedback, highlighting the connections between new ideas and classical topics. These results allow us to utilize known information theoretic converse theorems to establish fundamental bounds on passive methods of inference in the physical world.

On the algorithmic side, the formulation allows us to first unify 3D scene reconstruction and generative rendering as perception-aware generalization of decoding the physical world. Reviewing advances in conditional diffusion and video generation together with 3D Gaussian splatting, we illustrate the power of the framework algorithmically. In particular, we quantify multi-resolution uncertainty to not only reason about value of information across sensing modalities, but also to provide a recipe for actively acquiring information and evidence across time and space. We illustrate this through an example where we integrate RF and RGB modality for 3D scene rendering, video generation, and high-fidelity active 4D world models.