LiDAR Simulation for Training: Why Standard 3D Assets Break It

Key takeaways

•  Labelling is the bottleneck. One 3D frame costs six to ten times what a photo costs, and a production system needs millions.
• Simulation fixes volume, not accuracy. Detectors trained only on simulated scans have scored below 20%, against more than 70% on real data.
• The fault is in the 3D assets, not the simulator. Off-the-shelf models describe how a surface looks, never how it reflects infrared.
• CARLA's LiDAR ignores what surfaces are made of. Isaac Sim can read it, but only if someone typed it in. Marketplace assets never include it.
• Physicl ships that layer. Every surface tagged with composition, weight, grip and reflectivity at sensor wavelengths, 98% ready to simulate, labels included.

If you have trained a LiDAR detector, you know the pattern. The architecture is settled and the tooling works. What you do not have is enough labelled point clouds, and the ones you need most are the ones your fleet almost never sees.

So you turn to LiDAR simulation, and the scans come back wrong. Not obviously wrong. A black car that should fade out at 60 metres stays solid to 200. Glass that should let the pulse through bounces it back. The model trains happily, then misses dark vehicles on the road.

The simulator is not the problem. The 3D assets are.

Physicl's private beta will open soon. Request access and get physics-tagged assets built for your sensor stack.

The Core Bottleneck in Modern LiDAR Training

A LiDAR sweeps its surroundings with infrared pulses and records where each one bounced. The result is a point cloud: a few hundred thousand dots per frame, no colour, no texture, just positions and return strength.

Training a car or robot to read one means marking every vehicle, pedestrian and obstacle in it.

The models are not the constraint. PointPillars and CenterPoint, the designs most teams build on, are open and well understood. The point clouds are the constraint, and both collecting and labelling them scale badly.

Why Manual Point Cloud Labeling Fails to Scale

A 128-beam sensor produces around a quarter of a million points per frame, ten to twenty times a second, roughly five million points a second. One hour of driving yields tens of thousands of frames. A production system needs millions.

Each frame costs six to ten times what a photo costs. Someone draws a 3D box around every object, setting position, size and direction, in a thin scatter of dots with objects half hidden behind each other. A pedestrian at 80 metres might be 10 to 20 dots.

More labellers do not help:

• Errors grow with the team. Two people given the same frame draw different boxes pointing different ways. That disagreement caps model accuracy, and a wrong direction corrupts the prediction of where the object goes next.
• Cost is linear, accuracy is not. Each doubling of the labelled set buys less than the last. Budget runs out before the rare cases are covered.
• Rare cases cannot be bought. No fleet finds a mattress in the middle lane, in rain, at dusk, ten thousand times.

Pre-labelling software saves up to 68% of the time, but it inherits the blind spots of the model behind it.

The Missing Piece in Synthetic LiDAR Training Datasets

Simulation removes both limits. The simulator already knows every object's position, size and class, so labels are free and perfect. Rare cases become a request.

The catch is fidelity. Published results put the simulation-to-reality gap as wide as 50 percentage points: above 70% accuracy trained on real scans, below 20% trained on simulated ones.

Blame usually lands on the simulated sensor: beam spread, missing returns, noise. But a reading depends on the sensor and the surface. Tools have spent a decade on the sensor. The surface comes from a 3D library built for making pictures, which never needed it.

Why Standard 3D Models Break Synthetic LiDAR Training

Polygon count, texture layout, colour, glossiness. A marketplace asset describes how an object looks, not how it behaves in infrared, what it weighs or how much it grips. Enough for camera training. For LiDAR, it is the whole problem.

The Ray-Casting Problem: What a Surface Is Made Of vs. How It Looks

LiDAR simulation means tracing each pulse through the scene and working out what returns. That tracing is ray-casting.

A bounce registers only if the surface is reflective enough at that wavelength to clear the sensor's detection threshold. Below it, the object is not there. And the payoff is thin: nine times the reflectivity buys 1.73 times the range.

At 905 nm, the wavelength most car LiDARs use, real surfaces span two orders of magnitude:

‍

Source: CalibVision, LiDAR Detection Range vs Reflectance. Signage exceeds 100% because it sends light back where it came from instead of scattering it.

Most LiDAR simulation throws that spread away.

CARLA's LiDAR ignores what a surface is made of. CARLA does model materials elsewhere: Unreal's physical materials drive tyre grip and vehicle dynamics. Its ray-cast LiDAR is a different matter. Intensity is exp(-a·d), atmospheric attenuation times distance, with no material term at all. A black sedan and a white van at the same distance come back identical.

Isaac Sim, NVIDIA's robotics simulator, can use material, if someone put it there. NVIDIA states that "Lidar return behavior depends on material properties (for example, emissivity, reflectivity)". Those values live in a second material, typed in by hand on top of the appearance one. Marketplace assets ship appearance and nothing else, so every surface answers the sensor identically.

The capability exists. The data does not.

Wrong Reflections, Real Failures on the Road

Untagged assets do not produce broken scans. They produce believable ones, which pass visual review and train models that fail predictably.

• Glass. Infrared passes through car windows, so a real sensor sees the cabin as an empty gap. Untagged, it bounces back and the car is a solid block. The detector learns a shape that does not exist.
• Dark paint. Black paint reflects infrared badly, which is why researchers are engineering pigments that do better, one reaching 40.9% at 905 nm. A real dark car thins out then vanishes with distance. Untagged, it stays solid to the sensor's limit. Missing dark objects is usually the first field failure teams see.
• Reflective signage. Signs, number plates and hi-vis clothing send light straight back and overload many sensors. These are the strongest returns in a real scan. Untagged, they read as dull surfaces.
•  Wet and shiny surfaces. Standing water and polished metal bounce the pulse away. Untagged, the road stays evenly dense in the rain.

Each mismatch is invisible in a preview image and shows up only as a field failure, on exactly the cases the synthetic data was built to cover.

The usual fix is random variation: nudge signal strengths, drop points. That is noise on top of a wrong answer.

Accelerating LiDAR Training with Physicl.ai's Simulation-Ready Assets

Assets need to carry the physical properties the sensor model already knows how to read.

98% Sim-Ready: Nothing to Clean Up Before You Train

Physicl assets arrive 98% ready to simulate:

• Geometry sealed and checked, correct size, collision shapes and origin points set.
•  Structure consistent: moving parts split out and hinged, one convention for orientation and units, built on OpenUSD, the scene format Isaac Sim and Omniverse read natively.
• Materials carrying real physical values, not appearance settings.

The cost being removed is engineering hours, not licence fees. Marketplace sourcing means weeks per environment rebuilding geometry, re-rigging, re-scaling and typing in material values, done by simulation engineers rather than 3D artists. And because those libraries pool thousands of artists with different habits, every asset needs checking on its own.

Physicl rebuilds messy input to one standard, deriving what each surface is made of instead of guessing from texture. The result drops into Isaac Sim, Isaac Lab, MuJoCo or an in-house engine with no clean-up.

Physics Tagging: Perfect Labels That Work With Any Sensor

Every surface carries its composition, weight, grip and reflectivity at the wavelengths sensors use rather than the ones eyes use.

That last property is why one asset serves every sensor. Described by physics rather than by one simulator's settings, it returns correctly for a 905 nm unit, a 1550 nm long-range unit and the newer FMCW designs that measure speed as well as distance.

Labels come out of the same pass, exact: a class and ID on every point, 3D boxes for every object including fully hidden ones, plus distance, surface angle and material. No queue, no labeller disagreement, no accumulated direction error.

Scenes are built by code, so one checked environment reshuffles into thousands of variants across layout, clutter, wear, light and weather, relabelled each time.

Collection becomes generation.

Keeping Camera, Radar and LiDAR Training in Sync

A fused system is only as good as how well its training data lines up across sensors. Usually the three streams come from different tools on different versions of the scene, then get aligned afterwards, and small errors in position and timing corrupt the exact connections the model is learning.

Tagged assets remove that step. All three come from one scene with one set of materials, where the same property drives three effects:

• Camera reads colour and glossiness, producing an image with every pixel labelled.
•  LiDAR reads infrared reflectivity and shape, returning points with correct signal strength.
• Radar reads how the material carries and blocks radio waves. A metal barrier stands out on radar and barely registers on LiDAR; a plastic bumper does the reverse.

Sensor positions, timing and object identity are correct by construction, not by calibration. Disagreement cases become deliberate: fog that blinds the LiDAR while radar holds, or reflective clutter that overloads the LiDAR while the camera reads the sign perfectly. No fleet collects enough of those.

Why Tagged Data Survives the Next Sensor Generation

Sensor hardware keeps turning over. Wavelengths shift, scan patterns change, new designs replace old ones. Each change devalues a dataset collected on the previous generation, and a replacement costs quarters and millions.

Tagged data describes the world, not the sensor, so a new sensor means re-running the render. A collection programme becomes a compute job.

This extends past cars. A robot planning a grasp needs weight, grip and balance point, and one trained where every object weighs the same will struggle on real hardware. Reflectivity, weight and grip come from the same pass, so LiDAR training on tagged assets also builds the foundation for manipulation, navigation and embodied AI.

Physical AI is held back by data, not models. What the models lack is a description of the world accurate at the level the sensors measure. That is infrastructure, built once, not a problem every team solves alone.

Request access to the Private Beta, or talk to the team about physics-tagged assets for your LiDAR pipeline.

‍

‍