🆕 [2026-01-16] MetamerGen released — we upgraded the model to generate images that are metameric to human scene understanding (paper accepted at ICLR 2026), more details are here.
[2025] Seen2Scene released — the first iteration, introduced at CCN 2025, more details are here.
Ritik Raina1 · Abe Leite1 · Alexandros Graikos1 · Seoyoung Ahn2 · Dimitris Samaras1 · Gregory J. Zelinsky1
1Stony Brook University 2UC Berkeley
[MetamerGen Paper (ICLR 2026)] [Seen2Scene Paper (CCN 2025)] [BibTeX]
A family of models that reconstruct a person's understanding of a scene from their eye movements — turning sparse, foveated visual input into a complete generated image. Human vision blends low-resolution information from the periphery with sparse, high-resolution detail from where the eyes fixate.
Every model is a lightweight adapter over a frozen Stable Diffusion v1.5, conditioned on features from a frozen DINOv2 ViT-B/14 (reg4) encoder and injected through IP-Adapter cross-attention — only the small projection and adapter modules are trained.
The first iteration: three variants that isolate how foveal detail and scene gist each contribute to scene understanding.
| Model | Conditioning | Download |
|---|---|---|
| Fixation-only | DINOv2 patch tokens at fixated locations | adapter weights |
| Fixation + gist | fixated patch tokens + global cls/register gist tokens | coming soon |
| Gist-only | global gist tokens only (cls + 4 register = 5 tokens) | adapter weights |
The upgrade: generates images that are metameric to human scene understanding from a dual-stream foveated representation.
| Model | Conditioning | Download |
|---|---|---|
| MetamerGen | foveal: sharp patch tokens at fixated locations · peripheral: patch tokens from a blurred image (no gist tokens) | coming soon |
The code expects a Linux environment with a CUDA GPU (reference setup: Python 3.12, PyTorch 2.4.1, CUDA 12.1).
Quick start (Recommended) — clone the repository, then create the conda environment and download the DINOv2 encoder weights in one step:
git clone https://github.com/rainarit/seen2scene
cd seen2scene
bash setup.sh
conda activate eye-movement-scene-genconda — create and activate the environment from the provided definition:
conda env create -f environment.yml
conda activate eye-movement-scene-genpip — install a torch build matching your CUDA first (see the file header), then:
pip install -r requirements.txtsetup.sh fetches the frozen DINOv2 encoder into weights/. The trained
adapter checkpoints are separate — download the ones you want from the
Models table (GitHub Releases) into weights/:
curl -L -o weights/seen2scene_fixation_only.bin \
https://github.com/rainarit/seen2scene/releases/download/seen2scene-v1.0/seen2scene_fixation_only.binStable Diffusion v1.5 is fetched automatically from the Hugging Face Hub on first use (no manual download needed).
pipeline.py provides Seen2ScenePipeline, a diffusers-style
interface to every model in the family: pick a model with a string, feed an
image and the fixation (x, y) pixel locations of a scanpath, and get the
generated scene back.
from pipeline import Seen2ScenePipeline
pipe = Seen2ScenePipeline.from_pretrained(
model_type="fixation-only", # "fixation-only" | "fixation-gist" | "gist-only" | "metamergen"
adapter_path="weights/seen2scene_fixation_only.bin",
image_encoder_path="weights/dinov2_vitb14_reg4_pretrain.pth",
device="cuda",
)
scene = pipe(
image="my_photo.jpg", # PIL image, file path, or URL
fixations=[(310, 140), (95, 300), (420, 260)], # scanpath, (x, y) pixels in the input image
num_inference_steps=50,
guidance_scale=7.5,
seed=42,
)
scene.save("generated.png")Fixations are given in the input image's own pixel coordinates; the pipeline
maps them through the same aspect-preserving resize used at training time, so
each fixation selects the DINOv2 patch that actually contains it. gist-only
needs no fixations, and metamergen additionally accepts a peripheral_level
controlling how much the peripheral stream is blurred. Run it from the repo
root (or add the repo root and common/ to PYTHONPATH).
Two notebooks are provided to explore the models on any online image:
- Seen2Scene demo — sample random fixations, display the RGB PCA of the DINOv2 conditioning features (fixated patch tokens + gist tokens), and generate scenes with all three Seen2Scene models
- MetamerGen demo — build the sharp (foveal) and blurred (peripheral) views, display the RGB PCA of both streams, and generate a scene with MetamerGen
| Resource | Where | Used for |
|---|---|---|
| DINOv2 ViT-B/14 (reg4) | downloaded by setup.sh to weights/, or direct link |
frozen image encoder |
| Stable Diffusion v1.5 | auto-fetched from the HF Hub | frozen base model |
COCO train2017/ |
cocodataset.org | training images |
| COCO-FreeView fixations | official project page | evaluation (real human fixations) |
runwayml/stable-diffusion-v1-5 repo was removed from
the Hugging Face Hub; use the mirror stable-diffusion-v1-5/stable-diffusion-v1-5.
Set DATASET_PATH / DATASET_JSON_PATH inside the launcher scripts to your
local paths.
Edit the paths at the top of a launcher, then run it from the repo root. Only the adapter is trained; the base models stay frozen.
# Seen2Scene
bash seen2scene/scripts/train_fixation_only.sh
bash seen2scene/scripts/train_fixation_gist.sh
bash seen2scene/scripts/train_gist_only.sh
# MetamerGen
bash metamergen/scripts/train_metamergen.shCheckpoints are saved to runs/<model>_<timestamp>/. Metrics log to Weights &
Biases (wandb login, or set WANDB_MODE=offline).
The COCO-FreeView freeview evaluation (CLIP + DreamSim) belongs to Seen2Scene: it generates scenes for held-out human viewings and scores fidelity against the originals.
bash seen2scene/scripts/eval_gist_only.sh # gist-only
bash seen2scene/scripts/eval_freeview.sh # fixation-only / fixation+gistAccelerate saves sharded checkpoints; convert one to a single ip_adapter.bin
first (the gist eval does this automatically via
seen2scene/convert_gist_checkpoint.py; see also seen2scene/convert_checkpoint.py).
ℹ️ MetamerGen is assessed with the behavioral same/different metamer experiment described in its paper, not the freeview automated eval.
.
├── seen2scene/ # CCN 2025: 3 models + freeview eval
│ ├── train_fixation_only.py
│ ├── train_fixation_gist.py
│ ├── train_gist_only.py
│ ├── eval_gist_only.py # DreamSim/CLIP eval (gist-only)
│ ├── eval_freeview.py # DreamSim/CLIP eval (fixation / fixation+gist)
│ ├── eval_freeview_fov_per.py # DreamSim/CLIP eval (foveal+peripheral)
│ ├── convert_checkpoint.py, convert_gist_checkpoint.py
│ └── scripts/ # ready-to-edit train/eval launchers
├── metamergen/ # ICLR 2026: images metameric to human scene understanding
│ ├── train_metamergen.py
│ └── scripts/
├── pipeline.py # Seen2ScenePipeline: image + fixation scanpath -> generated scene
├── notebooks/ # interactive demos (PCA of features + generations)
├── common/ # shared datasets + eval metrics
│ ├── COCO_dataset.py # training data (foveation simulated on COCO)
│ ├── COCOFreeView_dataset.py # eval data (real human fixations)
│ └── eval_utils.py # DreamSim / CLIP metrics
├── dinov2/ # vendored DINOv2 encoder (code only)
├── ip_adapter/ # vendored IP-Adapter (Resampler, attn processors)
├── environment.yml # conda environment (recommended)
├── requirements.txt # pip alternative
└── setup.sh # create env + download DINOv2 weights
We are grateful to the open-source projects this work builds on:
DINOv2 (image encoder),
IP-Adapter (conditioning),
🤗 diffusers / Stable Diffusion v1.5,
and DreamSim (perceptual metric).
The vendored dinov2/ and ip_adapter/ directories retain their upstream licenses.
This project's code is released under the Apache License 2.0. The
vendored dinov2/ and ip_adapter/ directories are also Apache-2.0 and keep
their upstream copyrights (see NOTICE and their LICENSE files).
Stable Diffusion v1.5, which the models build on, is downloaded separately and is governed by the CreativeML OpenRAIL-M license — please comply with its use-based restrictions.
If you find this repository useful, please consider giving a star ⭐ and citing the paper(s) for the model(s) you use:
@inproceedings{raina2025seen2scene,
title={Seen2Scene: a generative model of fixation-by-fixation scene understanding},
author={Raina, Ritik and Leite, Abe and Graikos, Alexandros and Ahn, Seoyoung and Zelinsky, Gregory J.},
booktitle={Conference on Cognitive Computational Neuroscience (CCN)},
year={2025}
}
@article{raina2026metamergen,
title={Generating metamers of human scene understanding},
author={Raina, Ritik and Leite, Abe and Graikos, Alexandros and Ahn, Seoyoung and Samaras, Dimitris and Zelinsky, Gregory J.},
journal={arXiv:2601.11675},
year={2026}
}