# Datasheet for TLT12 Following the structure of *Datasheets for Datasets* (Gebru et al., 2018). --- ## Motivation **For what purpose was the dataset created?** To study latent variable learning in predictive self-supervised learning under known ground truth. Standard image datasets force researchers to *assume* which nuisance factors a representation should discard; TLT12 renders scenes from an explicit 10-dimensional latent state and ships that state with every frame, so invariance and equivariance can be measured against the actual generative factors rather than proxies. The temporal structure, one or a few factors sweeping smoothly while the rest are held fixed, is what distinguishes it from per-image latent datasets such as 3DIEBench. **Who created the dataset and on whose behalf?** Friedemann Zenke, FMI for the study *Understanding neural circuit principles for representation learning through joint-embedding predictive architectures* (eLife, 2026; [10.7554/eLife.110895.1](https://doi.org/10.7554/eLife.110895.1)). **Who funded the creation of the dataset?** The project received funding from the Swiss National Science Foundation (PCEFP3_202981) and the Novartis Research Foundation. --- ## Composition **What do the instances represent?** One instance is a *sequence*: 32 rendered RGB frames at 128 × 128 showing a single ShapeNet object in a simple scene (ground plane, fixed sun light, one coloured spot light), together with the full latent state of every frame. **How many instances are there?** 12 categories × 10 objects × 1000 sequences ≈ 120,000 sequences, ≈ 3.84 M frames. Exact counts per split and per category are in `stats.json`. **Is the dataset a sample or a complete set?** A sample. The 12 categories are a subset of ShapeNetCore.v2's 55, and 10 objects per category are sampled from each. Category selection favours objects with clear pose structure and a mix of rigid, near-symmetric, and asymmetric shapes: airplane, cap, car, chair, earphone, guitar, laptop, motorcycle, mug, piano, telephone, watercraft. Object sampling within each category was random with a fixed seed; the exact set is recorded in `stats.json` and reproducible via `generation/tlt12_commands.sh`. **What data does each instance consist of?** | Field | Type | Description | |---|---|---| | `frames` | `[32, 3, 128, 128]` float | RGB, values in `[0, 1]` (JPEG on disk) | | `latents` | `[32, 10]` float32 | full latent state per frame | | `base_latent` | `[10]` float32 | the frozen scene state for this sequence | | `traversal_velocities` | `[10]` float32 | `0` = frozen, `±v` = *initial* sweep rate | | `traversal_factors` | `list[int]` | indices of the factors that vary | | `synset_id`, `obj_id` | `str` | ShapeNet category and object instance | **Is there a label or target?** Several, depending on the task: the category (`synset_id`, mapped to a contiguous index by `synset_labels.json`), the object identity (`obj_id`), and the continuous latent state (`latents`), which supports regression-style equivariance probes. **Is any information missing?** The source 3D meshes are not included — ShapeNet's terms do not permit redistributing them. Depth maps, segmentation masks, and camera intrinsics were not rendered. The camera is fixed and identical across all sequences. **Are relationships between instances made explicit?** Yes. Sequences from the same object share an `obj_id`; sequences from the same category share a `synset_id`. Note that these relationships **cross the split boundary** — see Limitations. **Are there recommended data splits?** `train` / `val` / `test` at 80 / 10 / 10, shipped as separate shard subdirectories, split by random shuffle of sequences with a fixed seed. See the caveat under Limitations. **Are there errors, sources of noise, or redundancies?** Rendering is deterministic given the latent state, so there is no sensor noise. Two known issues: 1. *Path-tracer sampling noise.* Frames are rendered at a finite Cycles sample count, leaving mild high-frequency noise that differs between frames of a sequence even where the scene is static. 2. *ShapeNet mesh defects.* Most ShapeNetCore.v2 meshes store every face twice on shared vertices, causing z-fighting patches that flicker as the object rotates. The generator removes these duplicate faces at load time; see `generation/provenance.json` for whether this release was rendered with the repair enabled. Some meshes additionally have inconsistent normals or missing textures upstream, which is not corrected. **Is the dataset self-contained?** Yes for use. Shards, metadata, loader, and licenses are all included; nothing is fetched at load time. Reproducing the dataset from scratch additionally requires ShapeNetCore.v2 and BlenderProc. **Does the dataset contain confidential, offensive, or personal data?** No. All content is synthetic renders of inanimate manufactured objects. No people, faces, text, or identifiable locations appear. --- ## Collection process **How was the data acquired?** Rendered, not collected. Blender Cycles via BlenderProc, driven by `generate_traversals.py` from the generator repository. Each sequence draws a base latent, selects factors to vary, and renders 32 frames along that trajectory. **What mechanisms or procedures were used?** See `generation/tlt12_commands.sh` for the exact command sequence and `generation/provenance.json` for render settings and the generator commit. Rendering ran as a 60-task SLURM array on GPU nodes. **Over what timeframe was the data collected?** Rendering took approximately 500 GPU hours on a cluster containing Nvidia V100 and A100 GPUs. **Were ethical review processes conducted?** Not applicable — no human subjects, no personal data. --- ## Preprocessing / cleaning / labeling Frames are written as JPEG at Blender's default quality and are packed into WebDataset shards **without re-encoding**, so the released bytes are exactly the renderer's output. Latents are stored as float32. Duplicate-face removal is a mesh repair applied *before* rendering, not a post-process on images. No cropping, resizing, colour correction, or filtering is applied after rendering. The raw per-frame directory tree produced by the renderer is not distributed; the shards contain the identical JPEG bytes in a form that streams efficiently. --- ## Uses **What tasks is the dataset suited to?** Self-supervised representation learning with temporal prediction; measuring invariance and equivariance to individual generative factors; latent-factor regression; disentanglement evaluation; and — with the caveats below — object category classification as a downstream probe. **What should the dataset not be used for?** It is not a natural-image benchmark and results will not transfer directly to real photographs: lighting is simple, backgrounds are uniform, and the camera never moves. With 120 object instances total, it is unsuited to studies of object-level generalisation or to any claim requiring instance diversity. Commercial use is excluded by the license. **Is there anything that might affect future uses?** The category set is small and skewed toward rigid man-made objects, inheriting ShapeNet's composition biases. Nothing in the dataset targets or describes people. --- ## Limitations 1. **Splits share objects.** The split shuffles sequences, not objects, so all 120 objects appear in train, val, and test. Held-out metrics therefore measure generalisation to unseen *latent configurations*, not unseen objects. Regroup by `obj_id` if you need object-level held-out evaluation. 2. **Few object instances.** 10 per category; the design budget went into sequence count rather than instance diversity. 3. **Fixed camera and simple scenes.** No occlusion, no clutter, no background variation beyond floor hue. 4. **Rendering differences from 3DIEBench.** Same source corpus, but different latent parameterisation, temporal structure, and mesh handling. Numbers are not directly comparable across the two datasets. --- ## Distribution **How is the dataset distributed?** As WebDataset tar shards. Zenodo and institutional hosting. **When?** Release: September, 2026. **Under what license?** CC BY-NC 4.0 for the data (`LICENSE-DATA`), GPL v3.0 for the accompanying code (`LICENSE-CODE`). Use is additionally subject to the [ShapeNet Terms of Use](https://shapenet.org/terms), as TLT12 is a derived work of ShapeNetCore.v2. **Are there export controls or regulatory restrictions?** None known. --- ## Maintenance **Who maintains the dataset?** Friedemann Zenke and Ashena Gorgan Mohammadi **Will it be updated?** TLT12 is a pinned instance and will not change. Corrections, if any, will be published as a new version with a new DOI rather than by editing these files in place; `generation/provenance.json` carries the version number. **How can others extend or build on it?** The generator is open source at and supports arbitrary categories, resolutions, sequence lengths, factor subsets, and traversal velocities. Building a variant requires a copy of ShapeNetCore.v2 (free registration at [shapenet.org](https://shapenet.org/)).