We provide TLT12 (/tilted twelve/), a Temporal Latent Traversal dataset for evaluating structure learning and untangling of static and dynamic latent variables in predictive self-supervised learning models.

TLT12 is a rendered video dataset for studying latent variable discovery, both invariance and equivariance, in predictive self-supervised learning. Unlike datasets that sample a random pose per image, every sample here is a sequence in which one or more latent factors sweep smoothly across their range while the others stay fixed. The full generative latent state is shipped alongside every frame, so what a model should be invariant or equivariant to is known exactly rather than assumed.

The dataset is rendered from 12 object categories of ShapeNetCore.v2 using Blender Cycles via BlenderProc. It was introduced in Gorgan Mohammadi, Halvagal & Zenke (eLife, 2026).

Categories	12 ShapeNet synsets, 10 objects each
Sequence length	32 frames
Resolution	128 × 128 RGB
Latent factors	10, continuous, fully annotated per frame
Format	WebDataset tar shards
Splits	train / val / test (80 / 10 / 10 by sequence) + separate ood-test (held-out objects)

Authoritative per-split counts are in stats.json and in each split’s dataset_info.json.

Example traversals

Download

https://zenkelab.org/datasets/TLT12

Use download.sh to simplify the download.

Contents

TLT12/
├── README.md                 this file
├── download.sh               fetch the dataset (resumable, checksum-verified)
├── DATASHEET.md              datasheet: composition, collection, intended use, limitations
├── LICENSE                   CC BY-NC 4.0 — covers the shards, previews, and metadata
├── CITATION.cff              machine-readable citation
├── requirements.txt          dependencies for the provided loader
├── synset_labels.json        synset ID → category name and class index
├── SHA256SUMS                checksums for every file — also the download manifest
├── stats.json                per-split and per-category sequence counts, object list
├── wds_dataset.py            PyTorch loader for the shards
├── example_load.py           runnable example: load a batch, print it, save a GIF
├── verify_release.py         check your download is complete and intact
├── make_release_manifest.py  (for the maintainer) regenerate SHA256SUMS and stats.json
├── generation/
│   ├── tlt12_commands.sh     the exact pipeline that produced this release
│   └── provenance.json       generator version, render settings, packaging settings
├── examples/                 animated previews of representative sequences
└── shards/
    ├── train/      shard-*.tar.gz + dataset_info.json
    ├── val/        shard-*.tar.gz + dataset_info.json
    ├── test/       shard-*.tar.gz + dataset_info.json
    └── ood-test/   shard-*.tar.gz + dataset_info.json

---

Quick start

1. Download

TLT12 is distributed as loose WebDataset shards. If you have downloaded the entire dataset as an archive from Zenodo or elsewhere you can skip this step and directly go to Step 2. Otherwise you can download it from our server. Use download.sh to fetch the shards under SHA256SUMS. It pulls every file it lists, and verifies the result:

./download.sh --dest ./TLT12

It is safe to interrupt and re-run: files already present with the correct checksum are skipped without contacting the server, and a partially downloaded file is resumed rather than fetched again.

Try a small piece first. Each of val and test is about a tenth of the dataset, and a split download still includes the loader, docs, and metadata, so it is a complete working release:

./download.sh --split val --dest ./TLT12    # ~10% of the total
./download.sh --list                        # file list and exact total size
./download.sh --jobs 4                      # 4 parallel transfers

wget -c -r -np -nH --cut-dirs=1 <BASE_URL>/ works too if you prefer, but you will need to check sha256sum -c SHA256SUMS yourself afterwards.

2. Verify and load

cd TLT12
pip install -r requirements.txt

python verify_release.py                     # structure, checksums, decode test
python example_load.py --shards shards/train --gif sequence.gif

verify_release.py uses only the standard library, so it runs before you install anything. If it reports a checksum failure, re-run download.sh (it refetches only the bad files).

In your own code:

from wds_dataset import TraversalWebDataset, make_loader

ds     = TraversalWebDataset("shards/train/*.tar.gz")
loader = make_loader(ds, batch_size=16, num_workers=4)

for batch in loader:
    batch["frames"]                # FloatTensor [B, 32, 3, 128, 128], values in [0, 1]
    batch["latents"]               # FloatTensor [B, 32, 10]  — latent state per frame
    batch["base_latent"]           # FloatTensor [B, 10]      — the frozen background state
    batch["traversal_velocities"]  # FloatTensor [B, 10]      — 0 = frozen, ±v = sweep speed
    batch["synset_id"]             # list[str]  — ShapeNet category
    batch["obj_id"]                # list[str]  — ShapeNet object instance
    batch["traversal_factors"]     # list[list[int]] — which factors vary in this sequence

Two filters return new pipelines:

rotating = ds.filter_by_factor("rot_z")     # sequences where rot_z sweeps
mugs     = ds.filter_by_synset("03797390")

TraversalWebDataset(urls, transform=...) accepts a per-frame callable ([3, H, W] → [3, H, W]) if you want to plug in a torchvision augmentation pipeline.

Using it without PyTorch

The shards are ordinary compressed tar files, so nothing here is required. Each sample is one sequence, keyed by an 8-digit index:

00000042.frame_0000.jpg  …  00000042.frame_0031.jpg   raw JPEG bytes
00000042.latents.npy        [32, 10] float32
00000042.base_latent.npy    [10]     float32
00000042.velocities.npy     [10]     float32
00000042.meta.json          {"synset_id", "obj_id", "seq_idx", "traversal_factors"}

tar tf shards/train/shard-00000.tar | head will show you the layout directly (use tar tzf if the shards are .tar.gz — webdataset reads either without any change to your code).

---

The latent space

Each scene is described by 10 continuous factors. latents[t] gives the exact state of frame t.

Index	Name	Range	Meaning
0	rot_x	[−π/8, π/8]	object rotation about x
1	rot_y	[−π/8, π/8]	object rotation about y
2	rot_z	[−π, π]	object rotation about z (circular)
3	floor_hue	[0, 1]	hue of the ground plane (circular)
4	spot_theta	[0, π/4]	spot light elevation
5	spot_phi	[0, 2π]	spot light azimuth (circular)
6	spot_hue	[0, 1]	spot light hue (circular)
7	trans_x	[−0.5, 0.5]	object translation along x
8	trans_y	[−0.5, 0.5]	object translation along y
9	trans_z	[−0.5, 0.5]	object translation along z

Rotations are Tait-Bryan (XYZ extrinsic) Euler angles. The camera is fixed; apparent viewpoint change comes from object rotation.

How a sequence is built. A base_latent is drawn per sequence and holds the scene fixed. A subset of factors (listed in traversal_factors) is then swept across the sequence at the rate in traversal_velocities: 0 means frozen, ±v means it moves v × its full range in that direction. The practical consequence: within one sequence, the varying factors are the only thing that changes, which is what makes clean invariance and equivariance measurements possible.

At the edge of a factor’s range. Circular factors (rot_z, both hues, spot_phi) wrap, so those sequences tile seamlessly. The bounded factors (rot_x, rot_y, spot_theta, the three translations) instead reflect: the value turns around and travels back, like an elastic collision. Position stays continuous — there is no visual jump or freeze — but the instantaneous velocity changes sign.

This matters if you use traversal_velocities for equivariance targets, because it records only the velocity a sequence started with. Do not assume latents[t] == base_latent + t · v · span / 31; past a turning point the true motion is the other way. For rot_x and rot_y at |v| = 1 a bounce is the common case, not an edge case — a full-span sweep from an interior start point has to reach a boundary. latents is always exact, so take instantaneous velocity from the frames themselves:

d = batch["latents"][:, 1:] - batch["latents"][:, :-1]   # [B, 31, 10]

(For circular factors, wrap d into [−span/2, span/2] before using it.)

Category labels come from synset_labels.json, which maps each synset ID to a name and a contiguous class_index in [0, 12).

---

Which objects are in which split

Every split records its own membership, so you never have to scan the shards:

import json
info = json.load(open("shards/test/dataset_info.json"))

info["split_by"]    # 'sequence' or 'object' — what the held-out split measures
info["split_seed"]  # reproduces the partition
info["n_objects"]   # number of distinct objects in this split
info["objects"]     # {synset_id: [obj_id, ...]} — exactly which ones

stats.json carries the same lists for all splits at once, together with splits_are_object_disjoint and a pairwise object_overlap_between_splits count — both measured from the shards themselves rather than copied from the metadata.

---

Known issues and differences from 3DIEBench

Splits share objects. The train/val/test split is a random shuffle of sequences, not of objects, so the same 120 object instances appear in all three splits. Held-out performance therefore measures generalisation to unseen latent configurations, not to unseen objects. If you need object-level held-out evaluation, use the ood-test split which contains ten different objects per synset.

Duplicated ShapeNet faces. Most ShapeNetCore.v2 meshes store every face twice on the same shared vertices. Under a path tracer every ray hit then ties on depth, and the winner flips with BVH traversal order as the object rotates — producing polygon-bounded patches that switch on and off between frames. This is a systematic property of the ShapeNet export (a corpus scan found ~98% of models affected, most of them at exactly 2× copies per face), and it is visible wherever the two copies carry different materials. The generator removes the duplicates at load time. See generation/provenance.json for whether this release was rendered with that repair enabled; it is the main respect in which these renders differ from 3DIEBench, which was produced from the same corpus without it.

Small object count. 10 objects per category is deliberate, the design budget goes into 1000 sequences per object rather than into instance diversity. Treat category-level classification results accordingly.

---

License

The dataset shards, previews, and metadata are released under CC BY-NC 4.0. See LICENSE following the non-commercial Terms of Use dictated by ShapeNet.

Code — The code to generate the dataset resides in a separate repository and released under GPL v3.0, inherited from generator Code underlying from Meta’s SIE release: https://github.com/fzenke/3D_latent_traversal_dataset_generator/

TLT12 is a derived work of ShapeNetCore.v2. The 3D models themselves are not redistributed here, but your use of these renders is additionally subject to the ShapeNet Terms of Use. Where the two licenses differ, the more restrictive condition applies.

---

Citation

If you use TLT12, please cite the paper that introduced it:

@article{gorganmohammadi2026jepa,
  title   = {Understanding neural circuit principles for representation learning
             through joint-embedding predictive architectures},
  author  = {Gorgan Mohammadi, Ashena and Halvagal, Manu Srinath and Zenke, Friedemann},
  journal = {eLife},
  year    = {2026},
  doi     = {10.7554/eLife.110895.1},
  url     = {https://doi.org/10.7554/eLife.110895.1},
}

Please also cite ShapeNet, and, since the data generation process was derived from it, SIE / 3DIEBench:

@article{chang2015shapenet,
  title   = {ShapeNet: An Information-Rich 3D Model Repository},
  author  = {Chang, Angel X. and Funkhouser, Thomas and Guibas, Leonidas and
             Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and
             Savva, Manolis and Song, Shuran and Su, Hao and Xiao, Jianxiong and
             Yi, Li and Yu, Fisher},
  journal = {arXiv preprint arXiv:1512.03012},
  year    = {2015}
}

@inproceedings{garrido2023sie,
  title     = {Self-supervised learning of Split Invariant Equivariant representations},
  author    = {Garrido, Quentin and Najman, Laurent and LeCun, Yann},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2023}
}

---

Generating your own variant

The generator is open source:

https://github.com/fzenke/3D_latent_traversal_dataset_generator

It renders arbitrary latent-traversal datasets from ShapeNet: different categories, resolutions, sequence lengths, factor subsets, and traversal velocities. generation/tlt12_commands.sh records the exact pipeline used for this release; generation/provenance.json pins the generator commit.

You will need your own copy of ShapeNetCore.v2, which requires free registration at shapenet.org.