A Mirror on Wheels: Building a Digital Twin of Every Used Car
How we're working to turn a few minutes of smartphone video into an interactive 3D model of the actual car being sold, defects included.
It's 11 p.m., and Claire is scrolling through photos of a used estate car parked at a dealership 300 km away. The price is right and the mileage is reasonable. But photo 14 shows a scratch on the rear door. Is it two centimetres or twenty? There's no picture of the engine bay, and the shot of the driver's seat is oddly framed. She can either take a day off and a train to find out, or move on to the next listing. Most buyers move on.
Key takeaways
- Photos and 360° spins show only what the seller chooses to show. They carry no geometry, no scale and no depth.
- Recent work in 3D reconstruction makes photorealistic, real-time 3D possible, but cars are among the hardest objects to capture.
- Our research program aims at an articulated, measurable digital twin of each real vehicle, captured with a smartphone.
- This is ongoing R&D. We target pilots with dealers in H1 2028 (estimate).
Why photos and 360° spins are no longer enough
Online car retail still runs mostly on photos. Photos are useful, but they are a curated story: the seller picks the angle, the light and what gets left out. A photo can't tell you how big a dent is, because there is no reliable scale and no depth.
360° walkarounds were a real step forward, and buyers respond to them. A Cars.com analysis of more than 400 dealers found that listings with 360° content generated 68% more leads submitted and 92% more saved vehicles [1]. Even a modest step towards spatial, explorable content changes buyer behaviour.
A 360° spin is still a sequence of pictures, though. You can't open the trunk, lean into the door frame or check how a chip on the hood relates to the panel around it. The car is shown, but there is nothing you can measure.
"A photo shows you a car. A digital twin should let you inspect one."
What the research world has achieved
The last six years have been remarkable for 3D computer vision.
In 2020, Neural Radiance Fields (NeRF) showed that a neural network could learn a continuous 3D representation of a scene from ordinary photos, then render it from viewpoints no camera ever occupied [2]. The results were stunningly realistic, but slow: rendering a single image could take seconds.
In 2023, researchers at Inria introduced 3D Gaussian Splatting [3]. Instead of a network you query point by point, a scene is represented by millions of small, soft, coloured "blobs" in space, which a graphics card can draw very quickly. Photorealistic 3D suddenly ran in real time, which opened the door to interactive experiences in a browser.
Researchers have also started applying these ideas to vehicles. Recent work reconstructs car exteriors as they drive through fixed scanning rigs with calibrated cameras and controlled lighting [4]. That's a powerful setup, but it needs dedicated infrastructure, and it captures the car closed.
Separately, a whole field studies articulated objects, meaning things with parts that move: drawers, laptops, cabinet doors. Recent methods can work out from images how a part opens and around which axis [5][6]. Most of this work is still tested on fairly simple household objects in lab conditions.
What none of these pieces does alone is produce a complete, openable, measurable twin of a real used car, captured on a forecourt by someone who isn't a 3D specialist.
The open scientific questions
This is the heart of our program. Each challenge below is an active research problem, not an engineering checklist.
1. A car is basically a mirror on wheels
Most 3D reconstruction methods assume a surface looks roughly the same from every angle. A car breaks that assumption everywhere. Metallic paint, chrome trim and glass reflect the sky, the buildings and the person filming. Move the camera and the reflection slides across the panel, and an algorithm can mistake that moving reflection for real geometry.
Picture trying to judge the shape of a spoon by looking only at the reflections on it. That's roughly what reconstruction software faces on a freshly polished bonnet. Telling the paint apart from what it reflects is one of the hardest problems we're working on.
2. Understanding how the car opens
A typical car has four or five doors, a trunk or tailgate and a hood, each with its own hinge, angle and range of motion. Lab research on articulated objects usually works with one moving part, controlled captures and a cooperative object.
We want a dealership employee to capture all of this in a few minutes with a phone. It's a bit like learning how a cupboard works from a short holiday video: you need to infer the hinge, the axis and how far it swings, consistently across the whole car.
3. Inside the cabin and under the hood
Exteriors are hard. Interiors and engine bays are harder: tight spaces, low and uneven light, dark plastics, complex shapes and lots of occlusion. Then the inside has to join the outside in one coherent model, so that opening the driver's door in the twin shows the actual door frame and cabin behind it, in the right place.
Think of mapping a house room by room, then making sure every corridor lines up when you put the floor plans together.
4. Measuring defects, not just spotting them
Detecting a scratch in a photo is a known problem. Measuring it is a different one. A single photo gives you neither scale nor depth: a small scratch close up and a large scratch far away can look identical.
We want every scratch, dent or chip placed at its exact position on the vehicle, with an estimate of its real surface area in cm², and with an honest confidence level attached. A measurement without a confidence level can mislead as easily as no measurement at all. Like a good expert appraiser, the system should be able to say "about 4 cm², fairly sure" or "unclear, have a look in person".
5. Smooth on an ordinary phone, in the browser
None of this matters if the buyer needs a gaming PC. The twin has to load quickly and run smoothly on a mid-range phone, inside a web page, over a regular mobile connection. Getting a model that is detailed, photorealistic, openable and light enough to deliver is a research problem in its own right.
Our research direction
We won't go into methods here, but we can share two guiding principles.
First, a car is not an unknown object. Unlike a random scene, its overall structure is largely known in advance: where doors sit, how hoods hinge, how body panels relate to each other. We aim to exploit that prior knowledge so the capture can stay simple even when the reconstruction problem is hard.
Second, defects are data. We don't treat a scratch as a visual detail to be shown. We treat it as a measurable record with a location, a size and a confidence. Over time, that structured data could feed reconditioning cost estimates and more transparent pricing.
"We're not trying to make the car look better. We're trying to make it look exactly as it is."
What it could change
For dealers
- Listings that stand out because they show the real car, not a stock render.
- Fewer wasted visits: buyers who travel have already seen the car's actual condition.
- A documented, measured condition record that can help limit disputes after the sale.
For garages
- A measured, archived record of vehicle condition at check-in and check-out.
- Clearer conversations with customers about what was there before and what was repaired.
For buyers
- Seeing the actual car, defects included, before travelling.
- Opening the doors, checking the cabin and looking at the engine bay from the sofa.
Closing
Back to Claire. She shouldn't have to choose between a wasted day and a blind purchase. The tools to close that gap are starting to exist in research labs, but turning them into something a dealership can use every day, on every car, is still a scientific challenge. That's the challenge we've chosen.
If you run a dealership or a garage and would like to help shape this research, or join our future pilot programme, we'd love to hear from you. Get in touch with the Steer AI team.
References
- Impel / Cars.com, 360° walkaround case study (analysis of 400+ dealers), 2025.
- B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV 2020. arxiv.org/abs/2003.08934
- B. Kerbl, G. Kopanas, T. Leimkühler, G. Drettakis, "3D Gaussian Splatting for Real-Time Radiance Field Rendering," ACM Transactions on Graphics (SIGGRAPH), 2023. repo-sam.inria.fr/fungraph/3d-gaussian-splatting
- Kulkarni et al., drive-through vehicle reconstruction, arXiv, 2026.
- SplArt: articulated object reconstruction with Gaussian Splatting, arXiv, 2025.
- ArtGS: articulated object reconstruction with Gaussian Splatting, arXiv, 2025.
