AI that can simulate and understand physics to invent and discover.

Mission

We are building AI that can simulate and understand physics to invent and discover.

As AI keeps improving and intelligence gets more abundant, the bottleneck is shifting from coming up with new ideas to executing and improving them. Automating lab work and physical experiments streamlines this a little but doesn’t actually remove the bottleneck. We are replacing the experiments with AI that can simulate and understand physics.

Having this capability helps both with the “what if” of an experiment, and more importantly, with getting directional feedback. A real-world experiment can tell you what happened, but not why or how to improve. Our AI can.

We train models that take the state of a physical system and predict how it will evolve over time for a wide range of physical phenomena. It turns out training those capabilities into the same model rather than a narrow per-task surrogate works better because we can learn across domains. The models don’t have to be perfect from the start. They just have to be good enough to give useful feedback to let the improvement happen. You can still check the results in the lab or confirm with your existing process in the end. But your R&D pipeline and self-improvement loop don’t have to wait for the lab anymore.

Approach

We are building universal physical AI models that understand the world’s physics. This understanding comes not just from a symbolic level, as we see in language models, but from being able to simulate different kinds of physics and solve a wide range of tasks. This is useful to many areas such as engineering simulation, design, robotics and intelligence grounding. While language models provide high-level ideas, they lack the relevant physical grounding to make sure the ideas are valid, and therefore, physical experiments or simulations are still necessary. In practice, these experiments are precisely the bottleneck. We are instead training AI models directly on physical data to make them intrinsically understand physics.

In addition to physical experiments and simulations being the bottleneck they also happen to have another rather annoying property: while they do provide the outcome of the simulation or experiment, it comes with no direction or feedback of any form that can be used to take deliberate steps towards a hill-climbing goal such as optimizing a design. This means that in addition to the very slow feedback loop inherent to such experiments, the feedback may not be useful and is only helpful towards making a next educated guess. Our AI models, using their physical understanding, are instead able to also provide direction of improvement. This can be run in a loop to directly optimize to achieve a goal. This will shift engineering and R&D overall from a painful manual process of educated guesses in a slow manual loop towards letting engineers set up constraints and goals the models will solve for automatically.

Why our approach is the right one

We have made two distinct choices: one is the choice between broad vs narrow models. The second is if language and symbolic reasoning is the correct abstraction for physical understanding or if the backbone for intelligence should be physical. We believe that broad universal models based on physical understanding are the correct path towards engineering and scientific intelligence.

This is a distinct difference from using LLMs that lack physical grounding for design and discovery. As AI agents with, at best, high-level intuition, LLMs can deploy existing simulation tools to try to make judgements about a scenario. However, separating the intelligence layer from physical simulation creates a gap in both accuracy and the ability to get seamless feedback for continuous improvements. LLMs are limited in the same way as traditional R&D: they lack directional signals for goal optimization, and face an explicit simulation and execution bottleneck. Training models to intrinsically understand and simulate physics directly addresses this bottleneck.

Our approach may initially appear to be adjacent to so-called “world models” which generally limit themselves to plausible and good-looking visual appearance. We believe they are taking a number of shortcuts that may be acceptable to entertainment customers or simple navigation tasks, but this limits their usefulness for engineering. Some of these shortcuts will also make it harder to cover all rare edge cases. Some “world models” completely remove the time component, and therefore, most physical behavior. Even the ones that do consider time often focus only on what can be seen and usually compress that into a 2D spatial representation, as in video, which inherently lacks the relevant physical details for engineering and scientific tasks. Many of these models also run in an auto-regressive way, meaning they predict one step at a time, making errors compound.

Going for a full 4D (3D space + time) representation doesn’t just allow us to achieve the necessary quality required for engineering, but also simplifies our training process and reduces data collection needs: we can directly optimize for the adherence to known physical laws and constraints rather than being limited to sparse signal of possible observed consequences that may only be partially available in video data.

LLMs have proven that training one large model for a wide variety of tasks is not just successful at solving them, but also outperforms custom narrower models targeted towards a given task. We have confirmed this to be true for our physical AI models solving tasks that require the knowledge of physics: a model trained on a multitude of different physics doesn't just solve the tasks but outperforms models trained on those individual tasks at equal model size. Narrow models also uniquely face the problem of data availability. In industrial settings there are occasional pockets of abundant data that lead to flashy prototypes, but those usually happen to be far from the actual area of interest. In areas of interest the data points available are commonly sparse enough that they are barely sufficient for fine tuning of a universal model (should the model need it), but certainly not sufficient for a self-standing narrow surrogate. Training broad models also allows us to engineer a curriculum across data quality and resolution to reduce reliance on the most expensive sources.

Scale

To date we have completed hundreds of pre-training runs of model sizes up to a Trillion parameters. We have additionally performed scaling tests and confirmed our architecture to scale seamlessly with our largest training experiment at 35T parameters only limited by the size of cluster available to us at the time.

More important than the size of the model itself is the scale of context, meaning how much information can be processed at a time. For language, context is 1-dimensional, a sequence of text tokens. In our case the context needs to cover both three dimensions in space plus one in time, so 4 dimensions in total. For example 1000 points in each space dimension over 1000 time steps multiplies out to 1 Trillion. Frontier LLMs are able to handle up to a million in context or sometimes 10 million in experimental settings, which is far from sufficient for physical simulation and understanding. Our architecture is optimized for this, and has enabled us to achieve previously unheard of numbers: up to a Trillion context at training and exceeding 5 Trillion at inference without resorting to tricks like patching or sub-sampling. The output of a single inference sample at 5 Trillion context happens to be about 22 Terabytes.

Achieving this level of scale is not straightforward. Training approaches that work for LLMs don’t trivially translate to the scale we need so we had to develop our own. It starts with size: individual samples of data can be so large that they don’t fit into a single accelerator or even a full node. The same holds true for model layers, meaning we had to leave traditional strategies like FSDP behind and work on our own scalable sharding strategies.

In addition to scale we also want flexibility: not every use case needs this level of detail, and even for the most demanding use cases we do not need to use the same level of effort all the time. It would be ideal if we could dynamically choose the level of detail. A design process can start with a crude perspective that gets finer as more details are dialed in. Having to train and run a large number of different models for this would be extremely inconvenient. Our models happen to be resolution invariant, meaning they can be run at any resolution, both at training and inference time, making them ideal for modeling the physical world.

The video shows model outputs from one of our recent smaller (sub 100B) training runs. The 4D output is played back sequentially to make it accessible for human consumption. The model itself was producing the full rollout one shot. All different areas of physics come from the same model. The model has also not seen heat sink shapes at training time. We picked these areas of physics for diversity since they all come with their unique challenges and yet our model is able to learn to simulate all of them.

Self-Improvement

To understand where a journey leads it is helpful to both understand the current trajectory and what opportunities are available to further accelerate that trajectory. While we are proud of the scale and momentum we have already achieved, we are even more excited about some of our recent advances that are starting to pay off. Ultimately one would want both, some form of improvement loop for the model itself and another one for the ultimate task. That improvement loop should be better than brute force, especially if the brute force happens to be bottlenecked by something else like a lab experiment. We have both, a mechanism that lets us hill-climb model quality and one for discovery and invention.

Hill-climbing model quality: the public has recently seen substantial advances in AI for cybersecurity tasks. The reasons that this specific area happens to be a low hanging fruit for hill climbing are twofold: 1. tight feedback loops, meaning getting the result of an attempt does not require long-running lab experiments but can be had in a few seconds, and 2. clear signals of either success or failure. Most other areas do not have that luxury and are therefore at risk of slower advancement. Physical understanding, at least if it is approached the right way fortunately does. In fact physics may even have an advantage over cyber: The trick here is that it turns out physical laws for most areas we care about are known. They are known in a way that is traditionally inconvenient but sufficient for us: it is easy to test that a result satisfies the governing equation, which lets us drive our improvement loop, it is just hard to find a solution, until now. Even better: compared to cyber where you only get a discrete signal of "you made it" or "you didn't" that you need to design an extensive RL strategy around, in physics you get continuous scores and even a gradient that you can optimize for. Such physics feedback allows us to both overcome the reliance on training data alone, which can be expensive to obtain or generate, and drive the quality of the model to exceed it.

Hill-climbing invention and discovery itself: as intelligence and ideas get more abundant the bottleneck will shift elsewhere. Looping over real-world experiments or automating labwork only streamlines this a little without actually solving the bottleneck. And more critically, experiments can only tell you what happened, not why and how to improve. Our models both provide an alternative path to that bottleneck and improve it by giving directional feedback, completing the improvement loop. We also have a few ideas how we can self-improve and optimize the trajectory of the innovation loop itself but we can't talk about them just yet.