The age of empiricism in Physical AI
Necessity and sufficiency of Data in building generalizable autonomy
The year is 3025, and middle school history books have a chapter on AI and Robots. They remark on the 21st century as a brief period when mankind invented computers and subsequently discovered computational intelligence, and the excerpt from the book reads:
The era of computing began in mid 1900s when humanity invented computing and over the next 50 years refined the machine to become pervasive. Not long thereafter, principles of computational intelligence were discovered and it only took a few decades when intelligent machines both digital and physical became part of the fabric of the society. By 2100, humans were dependent on primitive robots for sustenance and there were more robots than people on the planet. These machines became the precursors to technologies of communication and life support, as we know them today.
It will be written with in same vein we write about game changers of last millennium, such as the printing press, the steam engine and the compound microscope.
We are in the midst of building next civilizational technology
“computational and physical intelligence”
and it will be built with data-driven empirical methods.
Humanity has frequently developed "know-how" before the "know-why”. We engineered technologies, mainly using observational understanding, such as metalworking, sailing ships, steam and internal combustion engines, and airplanes, all foundational to civilization, yet preceding a thorough understanding of their underlying sciences. Our empirical approach to building intelligence do not appear to be much different from prior technological conquests - driven by observation, developed using empirical methods, and iteratively refined over decades by a community rather than an individual.
Modern computing has only recently emerged as a tool for scientific discovery, yet it fundamentally redefines both the acquisition of "know-how" and the pursuit of "know-why." We can now create and collect planetary-scale data and perform comprehensive computations to compress it into a single model. This unprecedented speed and scale has upended our approach to scientific inquiry compared to any other point in human history.
In our quest to understand intelligence and “solve” robotics, I contend data is not merely beneficial, rather it is indispensable and foundational.
This necessity stems from several critical perspectives
1. Expressing Common-Sense in closed form
General-purpose intelligence, particularly in physical robots, is ambiguous, open-ended, and underspecified. Data offers guidance for achieving distributional similarity where a closed-form solution for common sense is absent.
Moreover, self-supervised learning—through observation or experience—allows for the encoding of concepts without manual programming. Lastly, learning with sparse yet critical feedback enhances robustness in observed behaviors and facilitates the discovery of novel ones.
2. (incorrect) Models may lead us astray
Accurate models foster innovation. Conversely, incomplete models can hinder the discovery of true optima, as these might be unrepresentable or unattainable with existing optimization tools.
Tom Mitchell in an excellent, and arguably timeless, monograph writes about the need for biases in generalization. A common challenge, and a criticism, of data-driven methods is generalization beyond the training set. A bias is, as Mitchell defines it, any basis for choosing one generalization over another, other than strict consistency with the observed training instances.
And Mitchell argued that unbiased generalization may be akin to retrieval from the training set. As such there is a tendency to bake in knowledge of the domain as an inductive bias. This knowledge can be included either through explicit changes in model hypothesis class, by structuring the data itself or in the optimization process. However, this intuition runs into an empirical problem, that is now popular as “the bitter lesson”.
As Andrew Wilson explains in his recent monograph, rather than restricting the hypothesis space to avoid overfitting, embracing a flexible hypothesis space with a soft preference for simpler solutions, that are consistent with the data, is often beneficial. As such in a specific domain, choice of strong restrictions through model or inductive biases may result in easy to interpret yet hard (or even impossible) to optimize problem formulations.
Our understanding of generalizable autonomy in robotics is currently insufficient to define even necessary conditions, let alone sufficient ones.
3. Computing uses abstractions, so should robotics!
Robotics represents the evolution of computing, moving beyond desktop, mobile, and cloud environments into the physical world. The science and engineering of computing has developed ideas & abstractions, and coverted these abstractions into systems — enabling us to build increasingly sophisticated computing machinery. This is evident in the hiding the complexity from a user of computing or a application developer.
Robotics with all of its messiness is akin to a new generation of computing. And lessons from previous eras of computing systems will serve us well as we build a new stack of Physical AI.
While data-driven approaches are often criticized for lacking elegance, the empirical process of data collection, creation, and curation is inherently a scientific endeavor. Moreover, the construction of complex technologies like foundation models involves intricate processes, despite their seemingly monolithic outputs. Therefore, we must apply principles of abstraction from other computing domains to build sophisticated yet well-behaved robotic systems.
4. Robotics needs a lightening rod!
Robotics remains a collection of disparate communities. The handbook of robotics is a 80+ chapter tome encompassing different sections from ranging from: Design. Sensing & Perception, Manipulation, Navigation, Applications, HRI!
Data-driven methods have successfully unified diverse problems in fields like Language Processing and computer vision. Foundation models now unify previously disparate problems such as text understanding, question answering, interactive chats, machine translation, among others. Similarly problems of detection, segmentation, semantic reasoning, and generation are now unified under a single class of models.
We must as a community explore adoption of the foundation model perspective, in order to unify many seemingly unrelated problems within the robotics community, given that all robots are governed by the same fundamental laws of physics.
Data-driven scaling is a necessary step in building artifacts of functional computational implementations of intelligent autonomy.
While necessary, the sufficiency of these methods continues to be in question! It may be a while before we can conclusively answer this question.
Research in machine learning — going all the way back to Philosophy (David Hume), Psychology (Goodman), and modern machine learning (Mitchell) and deep learning (Hinton, LeCun & Bengio) — has advocated for the need for biases, for without them the notion of generalization is ill-defined.
…make the inductive leap necessary to classify instances beyond observed...other sources of information, or biases for choosing one generalization over the other…
However, in our search for elegance and simplicity, at times the domain knowledge is more of a burden than it is of assistance. While our prior on the problem may be intuitive to us, the processes and algorithms we use to build large scale foundation models may not need or use such priors. We will however need to build systems that exhibit intelligence functionally and then create tools to analyze the why.
We as a community need to focus on building the engine rather than quibbling about the lack of laws of thermodynamics. They will crystallize in time, and it is very likely that artifacts such as robotics foundation models will help us in formulating and testing such laws.
Generalizable Autonomy for Physical Systems is an open-ended problem that requires us to find any necessary solution before we can develop a more elegant, interpretable, and efficient approach.
Our endeavors in creating data-driven technological artifacts are merely one step in enhancing our comprehension of the essence of intelligence itself.
Note: This is an addendum to the ICRA 2025 Keynote debate. Original slides