Models trained on synthetic data routinely fail on real near-infrared imaging (NIR)

01.06.2026|14:38 Uhr

The standard answer is “close the domain gap.” But the deeper question is: what is the model actually looking at, and what should it be looking at?

Our paper “Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery” is accepted at ECML PKDD 2026.

The diagnosis is that models trained on synthetic images tend to focus on local textures rather than scene geometry. Since synthetic textures differ from real ones, the models fail even when shapes and structures are essentially the same. 

Three contributions:
→ A generative augmentation pipeline that turns synthetic RGB scenes into realistic NIR-style training data, using a LoRA-fine-tuned latent diffusion model with multi-signal ControlNet conditioning. Structure stays intact, even for thin safety-critical classes like seat belts.

→ A Voronoi-based style diversification strategy that deliberately perturbs local textures while preserving scene geometry, thus pushing models toward shape-based representations.

→ A cue-decomposition framework that quantifies, for any segmentation model, how much it relies on shape versus texture. This is a measurement tool, not just a result.

The numbers: up to 28.4% of the synth2real domain gap closed on interior NIR scenes, up to 63.6% on exterior NIR scenes, without a single additional real annotation.

Why this matters: cue decomposition gives us a way to ask not just “does my model work?” but “what is it actually looking at?” For automotive perception, where regulation (EU 2019/2144) demands reliable driver and occupant monitoring, this question is no longer optional.

This work is part of Felix Stilger’s PhD at the TMDT, co-supervised with Aptiv as industry and scholarship partner. A collaboration across three German universities, Bergische Universität Wuppertal (TMDT), Heinrich-Heine-Universität Düsseldorf, and Universität Osnabrück plus Aptiv.

Congratulations to the team:
Felix Stillger, Ben Hamscher, Annika Mütze, Lukas Hahn, Kira Maag, Tobias Meisen.

We publish at the top venues of AI research and industrial application, and validate where it counts.