New paper from the lab led by Fabian Mikulasch.
📄 https://arxiv.org/abs/2609.37789
🌐 https://info-ldm.github.io/
Code: https://github.com/fmi-basel/identifiable-stochastic-nuisance
Why does predicting in latent space (JEPA, CPC, SimCLR…) work so well on messy data, with changing lighting, camera angles, and busy backgrounds? The usual answer: it can ignore nuisance. But that answer holds a conundrum.
The signals we care about themselves are stochastic. Both the stochastic signals and the nuisance variables make the next observation unpredictable. So how can a predictive model know what to keep and what to ignore? Prior identifiability theory covers either stochastic signal dynamics (without nuisance) or nuisance (but only deterministic signal dynamics). Video, sensors, and agents acting in the world have to deal with both at once.
We show common SSL methods can separate them when two mechanisms work together: predictive MI maximization retains all predictable information; latent distribution matching makes the retained signal identifiable. For Gaussian predictors, we prove recovery to an affine map.
We tested this on a MuJoCo hopper with non-deterministic dynamics and nuisance: random colors, lighting, camera, and noisy backgrounds. The latent predictive model’s representation allows to linearly decode the hopper’s pose and velocity.
Adding a pixel-reconstruction loss to the same model gives latents that partly encode nuisance and no longer allow decoding the state.
Finally, in ongoing work, the learned latent dynamics can be rolled out from a few observations (t<0) to recover the hopper’s signals, including uncertainty estimates.