Add Videoprism by MHRDYN7 · Pull Request #39895 · huggingface/transformers

MHRDYN7 · 2025-08-04T17:37:52Z

Fixes #39893. This pr adds the VideoPrism model by google deepmind. Original repo

MHRDYN7 · 2025-08-23T21:51:51Z

Summary of the code so far

The VideoPrismModel has been implemented using the modular code on top of Vivit and this is only a video encoder.
The code design is such that the same encoder code is reused as many times as possible.
Batches of video segments of shape (num_frames=16, H=288, W=288) are passed into the model and then the tubelet embedding class is used to convert each frame into hidden states of shape (256, 768) and therefore the whole input becomes (B*num_frame, 256, 768).
These spatial embeddings are passed into a spatial encoder
The outputs of the spatial encoder are reshaped to (B*256, num_frame, 768) and then passed into a temporal encoder
The attention function has an internal attention cap implemented in the modified eager_attention, not sure if this can be somehow used along with sdpa.
The VideoPrismClip model uses VideoPrismModel as a backbone for the video input, passes the embeddings through an auxiliary encoder then through an attention pooling layer.
There is also a standard text encoder that is called inside VideoPrismClip.
All the exact details from the original code have been extracted and correct tensors are being returned for both the models.

@qubvel I'd request your preliminary review on the current code structure. There are detailed comments with "# ?" for ease of review, these will be removed later on.

Todos:

Get started with the tests
Implement the video and text processors
Add code for the video classification model

** Please note that i am currently using the preprocessing utils from the original code in my convert_weights_to_hf script. The code uses mediapy, which uses lanczos interpolation for resizing videos by default. However, lanczos interpolation is not supported in torch yet and that's why we can't get the exact same outputs if a fast video processor is used.

** The videoprism team have not released the weights for the classification head.

** The weights released on hub are in npz format, safetensors need to be uploaded there.

…e fixed

…tokenizer

… exact values of jax.image.resize

qubvel

Hi @MHRDYN7, huge thanks for working on the model addition! You already did a great work, please see the comments to align it further to the transformers standards 🤗

qubvel · 2025-08-26T11:32:16Z

+@dataclass
+class TextEncoderOutput(ModelOutput):
+    """
+    Base class for text encoder outputs.
+    """
+
+    last_hidden_state: Optional[torch.FloatTensor] = None
+    hidden_states: Optional[tuple[torch.FloatTensor, ...]] = None
+    attentions: Optional[tuple[torch.FloatTensor, ...]] = None


This is just a common BaseModelOutput, no need to redefine it

qubvel · 2025-08-26T11:36:49Z

+        if self.mode == "spatial":
+            self.patch_embeddings = VideoPrismTubeletEmbeddings(config)
+            self.spatial_pos_emb = nn.Parameter(torch.zeros(1, self.pos_emb_shape[1] * self.pos_emb_shape[2], config.hidden_size))   # ? (1, 256, 768)
+
+        elif self.mode == "temporal":
+            self.temporal_pos_emb = nn.Parameter(torch.zeros(1, self.pos_emb_shape[0], config.hidden_size))  # ? (1, 16, 768)


Do we have checkpoint releases for both versions? Otherwise, please leave only one.

qubvel · 2025-08-26T11:38:13Z

+    def _interpolate_emb_2d(
+        self, emb: torch.Tensor, source_emb_shape: tuple[int, int], target_emb_shape: tuple[int, int]
+    ):


should be defined in interpolate_pos_encoding instead, no?

didn't use interpolate_pos_encoding because there are two different types interpolation for the pos embeds (spatial first then temporal). Now that the plan is to have two different embedding classes, I'll inherit from VivitEmbeddings for both and they both will have interpolate_pos_encoding method.

qubvel · 2025-08-26T11:40:41Z

+        with torch.no_grad():
+            self.layernorm_before.weight += nn.Parameter(
+                torch.ones(self.config.hidden_size)
+            )
+            self.layernorm_after.weight += nn.Parameter(
+                torch.ones(self.config.hidden_size)
+            )


That seems super strange to me. Why do we need this, is this correct? The operation is in-place, so after each forward pass, are we continuing to increase the weight?

I found it very strange too.

Here is the original code

https://github.com/google-deepmind/videoprism/blob/main/videoprism/layers.py#L182-L193

when direct_scale is set to False (this is always the case), +1 is added to the scale tensor of layernorm.

This code seems more like an attempt to ensure the layernorm scale factor is not zero during training. I can create an issue in the repo to confirm if they want this behavior during inference as well.

You are right that the hf code in the current form means that the layernorm scale will get increased by +1 for every iteration of forward pass. The jax code initializes the Layernorm class on the go just before it is called so this problem does not happen there. If this +1 portion is moved (from forward) to the init of the relevant class, then that does not work as I guess during creation of a model instance, the init methods are evoked with the initialized weights, and later the pretrained weights are placed and that's why +1 does not happen. I've been working with single forward passes of the model, so it's been fine, but this issue still needs to be resolved.

Ok, I suppose we must not modify weight inplace and define it as follows:

import torch import torch.nn as nn import torch.nn.functional as F class VideoPrismLayerNorm(nn.LayerNorm): def forward(self, input: torch.Tensor) -> torch.Tensor: return F.layer_norm( input, self.normalized_shape, self.weight + 1, self.bias, self.eps )

That should be equivalent, right?

Or in case +1 is only for initialization, we should not have it in modeling code, just in _init_weights method.
~~Please keep in mind that logits should match exactly (1e-3/1e-4) with the original implementation, and matching them should give you the right answer whether this addition is relevant~~

P.S. Just saw the message below 👍

alright, using F.layer_norm is a very good solution; I'll rename the weights and refactor the code. Also I got the confirmation from the videoprism team, scale = 0 for initialization, and the +1 is expected to be there.

qubvel · 2025-08-26T11:42:44Z

+        if mode == "spatial":
+            self.layer = nn.ModuleList([VideoPrismLayer(config) for _ in range(config.num_spatial_layers)])
+        elif mode == "temporal":
+            self.layer = nn.ModuleList([VideoPrismLayer(config) for _ in range(config.num_temporal_layers)])
+        elif mode == "auxiliary":
+            self.layer = nn.ModuleList([VideoPrismLayer(config) for _ in range(config.num_auxiliary_layers)])
+        elif mode == "unimodal":
+            self.layer = nn.ModuleList([VideoPrismLayer(config) for _ in range(config.num_unimodal_layers)])
+        else:
+            raise ValueError(f"Unknown mode: {mode}. Supported modes are: spatial, temporal, auxiliary and unimodal.")


I don't see any reason to split it into different if/else. We might have only one attribute num_layers and that's it

If we remove these conditionals, then we will need to set config.num_layers = config.num_spatial_layers, before instantiating the spatial encoder object and then config.num_layers = config.num_temporal_layers before temporal encoder in the model init. It's slightly less explicit, but, since you agreed with the change of config from python_gelu to relu in the init, I'll go ahead with this one too, and then the encoder can be directly used from vivit.

qubvel · 2025-08-26T11:50:45Z

+        with torch.no_grad():
+            self.layernorm.weight += nn.Parameter(torch.ones(self.config.hidden_size))


again, super strange

qubvel · 2025-08-26T11:52:20Z

+class PositionalEmbedding(nn.Module):
+    def __init__(self, config: VideoPrismConfig):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.min_timescale = 1
+        self.max_timescale = 10000
+
+    def forward(self, seq_length):
+        position = torch.arange(seq_length, dtype=torch.float32).unsqueeze(0)  # ? (1, seq_length)
+        num_timescales = self.hidden_size // 2
+
+        log_timescale_increment = math.log(
+            float(self.max_timescale) / float(self.min_timescale)  # ? log(10000/1) = ln(10000)
+        ) / torch.maximum(torch.tensor(num_timescales, dtype=torch.float32) - 1, torch.tensor(1))
+
+        inv_timescales = self.min_timescale * torch.exp(
+            torch.arange(num_timescales, dtype=torch.float32) * -log_timescale_increment
+        )
+
+        scaled_time = position.unsqueeze(-1) * inv_timescales.expand(1, 1, -1)
+
+        embs = torch.cat((torch.sin(scaled_time), torch.cos(scaled_time)), dim=-1)
+
+        return embs


align with the RoPE classes defined in transformers

qubvel · 2025-08-26T11:57:29Z

+        self.backbone = VideoPrismModel(config)
+        self.auxiliary_encoder = VideoPrismEncoder(config, mode="auxiliary")
+        self.contrastive_vision_pooler = VideoPrismMultiheadAttentionPoolingHead(config)
+        self.text_encoder = VideoPrismTextEncoder(config)


I assume we should have the following classes (similar to CLIP)

VideoPrismVideoModel

VideoPrismTextModel

please, align to this

qubvel · 2025-08-26T11:59:47Z

This should be refactored to follow mllama or dinov3_vit conversion script format. We should have a weights mapping dict and manipulate that

MHRDYN7 · 2025-08-27T10:58:28Z

The following notebooks are from the original repo and the expected values of the output tensors used to validate the HF implementation have been taken from here.
Video encoder model notebook
Video Text model notebook

Please note that the output tensors are slightly different for the larger models when a TPU is used. My expected tensor values are for the cpu.

Since the output of the final tensors (of HF code) matches that of the original code, the elements in the current code are aligned with the original implementation, despite that strange layernorm weight increase.

…refactor

…ctly

MHRDYN7 · 2026-06-19T15:28:13Z

Works again now 🥳

I'm loosening the tol for that test

github-actions · 2026-06-19T15:37:06Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, videoprism

vasqu · 2026-06-19T15:41:46Z

run-slow: videoprism

github-actions · 2026-06-19T15:43:04Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/videoprism"]
quantizations: []

MHRDYN7 · 2026-06-19T15:43:56Z

@vasqu do you remember the hf dataset repo for storing assets for docs? One image link needs to be replaced, it was already uploaded there

github-actions · 2026-06-19T15:54:42Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	3d08a2ad	workflow commit (merge commit)
PR	7cebc1dd	branch commit (from PR)
main	e7835fba	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

MHRDYN7 · 2026-06-19T15:58:36Z

Yes pretty much ready to merge, checking with slow CI. Can you update the checkpoints to use the official repos with the current revision?

I talked internally, it's handled and I will update the ckpt paths / revisions when they are merged/moved so should be all good 🤗

regarding this: everything uses the google repos except for the three test files where you will need to replace 'mhrdyn7' to 'google' via ctrl+F.

Slow tests passed, the very last commit simply contains the correct image link in doc, so no more run_slow needed

github-actions · 2026-06-19T15:59:04Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, videoprism

github-actions · 2026-06-19T16:01:17Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, videoprism

vasqu · 2026-06-19T16:24:42Z

@MHRDYN7 let's update the checkpoints on docs and tests, we can merge then imo

vasqu · 2026-06-19T17:24:25Z

I meant in this PR pls :D so to use the google ckpts with revision for all occurrences

github-actions · 2026-06-19T17:27:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, videoprism

MHRDYN7 · 2026-06-19T17:27:13Z

ok done. Had some confusion, but all good now. run_slow is needed, there was a stale tokenizer in a google ref, so just to be safe

vasqu · 2026-06-19T17:27:26Z

run-slow: videoprism

vasqu · 2026-06-19T17:27:36Z

Last sanity check 🙏

github-actions · 2026-06-19T17:28:58Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/videoprism"]
quantizations: []

github-actions · 2026-06-19T17:42:16Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	b5a5263c	workflow commit (merge commit)
PR	53cc84fb	branch commit (from PR)
main	95ed1b82	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu · 2026-06-19T17:49:16Z

Rerunning fast CI, don't understand why it failed tbh 🤔 otherwise will check next week and merge then

vasqu · 2026-06-19T18:04:05Z

Haya it worked!! Nice and gz on the merge @MHRDYN7 🫡

MHRDYN7 added 9 commits August 4, 2025 07:19

inference returns correct logits for base model

0a240be

modular code runs correctly

d50bbab

large backbone model added

ce461e4

a rough random forward pass

ac06354

converted and uploaded both lvt weights

b66cc62

updated lvt checkpoints, modeling code gives the correct logits for base

8e6766a

everything works fine with modular convertion at this point

34e1a28

standardization of the VideoPrismModel code

d702e88

Merge branch 'main' into videoprism

ec5ceeb

MHRDYN7 marked this pull request as ready for review August 23, 2025 21:08

github-actions Bot requested review from ArthurZucker and Rocketknight1 August 23, 2025 21:08

added base classes for model output holders, but signature needs to b…

46be807

…e fixed

ArthurZucker added the New model label Aug 25, 2025

correctly added slow videoprism tokenizer using modular on top of t5 …

87a67a8

…tokenizer

qubvel requested review from qubvel and removed request for ArthurZucker and Rocketknight1 August 25, 2025 10:54

MHRDYN7 added 2 commits August 25, 2025 19:34

temporal interpolation works fine but F.interpolate does not give the…

0d22817

… exact values of jax.image.resize

added fast tokenizer based on t5

e391f2c

qubvel reviewed Aug 26, 2025

View reviewed changes

MHRDYN7 mentioned this pull request Aug 27, 2025

Layernorm scale increase by +1 google-deepmind/videoprism#49

Open

MHRDYN7 added 6 commits August 28, 2025 17:04

Merge branch 'main' into videoprism

728a041

refactored code for VideoPrismModel runs fine on initialised weights

bee89bf

converted lvt base weights, but model outsputs are not correct after …

4452274

…refactor

perdimscale fixed and lvt base weights updated, inference works corre…

df17137

…ctly

refactored perdimscale + sinusoidal posemb to get correct lvt logits

c189df9

Merge branch 'main' into videoprism

6498fc3

new atol for video model

7cebc1d

correct image link

2626993

resolve

1a12490

vasqu reviewed Jun 19, 2026

View reviewed changes

Comment thread docs/source/en/model_doc/videoprism.md Outdated

vasqu reviewed Jun 19, 2026

View reviewed changes

Comment thread tests/models/videoprism/test_processing_videoprism.py Outdated

vasqu reviewed Jun 19, 2026

View reviewed changes

Comment thread tests/models/videoprism/test_tokenization_videoprism.py Outdated

vasqu reviewed Jun 19, 2026

View reviewed changes

Comment thread tests/models/videoprism/test_modeling_videoprism.py Outdated

correct checkpoints with revisions

53cc84f

vasqu added this pull request to the merge queue Jun 19, 2026

Merged via the queue into huggingface:main with commit 1048e9a Jun 19, 2026
190 of 195 checks passed

		with torch.no_grad():
		self.layernorm.weight += nn.Parameter(torch.ones(self.config.hidden_size))

Conversation

MHRDYN7 commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MHRDYN7 commented Aug 23, 2025

Summary of the code so far

Uh oh!

qubvel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qubvel Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qubvel Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MHRDYN7 Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MHRDYN7 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MHRDYN7 commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

vasqu commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

MHRDYN7 commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

CI Results

Commit Info

Uh oh!

MHRDYN7 commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vasqu commented Jun 19, 2026

Uh oh!

vasqu commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

MHRDYN7 commented Aug 4, 2025 •

edited

Loading

qubvel Aug 27, 2025 •

edited

Loading

qubvel Aug 27, 2025 •

edited

Loading

MHRDYN7 Aug 28, 2025 •

edited

Loading

MHRDYN7 commented Aug 27, 2025 •

edited

Loading