Meta Inverse Prompting for Video Generative Models

Yerin Jung

Jongheon Jeong

Korea University

NeurIPS 2026

Target video

Recovered prompt

“The video captures a person kiteboarding on a clear day at the beach. The person is wearing a black helmet and a life jacket, and they are holding onto a blue handle attached to a kite that is not visible in the frame. The person is riding a blue surfboard and is moving towards the right side of the frame. …”

Reconstruction

Abstract

Recent advances in video generative models have enabled flexible conditional generation and editing pipelines, such as text-to-video, image-to-video, and reference-guided synthesis. However, these workflows are inherently one-directional, mapping conditions to videos without providing a mechanism to invert this process. In complex media production settings, the ability to recover controllable textual conditions from a given video is highly desirable, yet remains underexplored.

While inverse prompting has been studied for image generation, directly extending these approaches to video is challenging due to the need to capture temporal dynamics beyond static appearance, as well as the substantial computational cost of per-video optimization. In this work, we propose the first practical framework for efficient video inverse prompting. Specifically, we formulate inverse prompting as a meta-learning problem and introduce Meta Video Inverse Prompting (MVIP), which learns a meta-prompt optimized over temporal dimensions to enable fast and effective prompt inference. Our approach amortizes the inversion process, eliminating the need for costly iterative optimization at test time.

Experimental results demonstrate that MVIP recovers higher-quality inverse prompts with significantly improved reconstruction fidelity compared to image-based inversion and iterative captioning baselines, while achieving substantial gains in efficiency.

Method

Existing workflows to access video generative models are inherently unidirectional: conditions can guide video synthesis, but there is no clear mechanism to invert this process. A creator may generate a video from a prompt and later wish to modify it, such as changing lighting or motion while preserving the scene. However, the original prompt is often unavailable or insufficient to reproduce the video, forcing users to rely on trial-and-error prompt engineering. Our goal is inverse prompting: recovering a reusable prompt from a target video, enabling both faithful video reconstruction and controllable prompt-based video editing.

Top: a target video is inverted into a prompt, and a video generator conditioned on that prompt reconstructs the video. Bottom: editing the recovered prompt (stationary to take off) and regenerating yields the edited video.

Prior inverse-prompting approaches do not scale well to video: gradient-based prompt optimization requires repeated forward and backward passes, which becomes prohibitively expensive for long video sequences, while captioning produces semantically plausible but insufficient prompts to support reconstruction. Our key idea is to unify the two within a meta-learning formulation: instead of optimizing prompts from scratch for each video, MVIP learns a shared prompt initialization across videos, which at inference time enables rapid prompt inference without costly per-video optimization, effectively behaving as a reconstruction-aware captioning mechanism.

Overview of MVIP: a pool of video tasks, an inner loop of reconstruction-guided adaptation with a frozen VLM and a frozen video generator, and an outer meta-update of the shared soft prompt.

MVIP meta-learns a shared soft-prompt initialization Φ for video inverse prompting across video tasks. Inner loop. A frozen VLM generates candidate prompts, and the candidate with the highest reconstruction score from a frozen video generator is selected as a pseudo-target for updating Φ. Outer loop. The task-adapted prompts update the shared initialization following Reptile. Inference. The meta-learned Φ is applied directly to unseen video segments; the VLM and the video generator stay frozen throughout.

Video reconstruction

Since text alone underconstrains video reconstruction, every method renders with the same frozen video generator conditioned on its prompt and the first and last frames of the target video, which specify how the video starts and ends.

Reference

MVIP Ours

FLF-only

PEZ [1]

Captioning [2]

Not applicable

The video depicts a desolate, barren landscape with a mysterious and eerie atmosphere. The scene is shrouded in a thick layer of fog or mist, creating an air of mystery and uncertainty. The terrain is rugged and rocky, with jagged formations jutting out from the ground, suggesting a harsh and unforgiving environment. In the distance, a large, humanoid figure emerges from the fog. The figure appears to be a creature with multiple limbs, possibly tentacles or appendages, giving it a somewhat alien or otherworldly appearance. Its form is somewhat obscured by the mist, but its presence is undeniable. The figure seems to be moving slowly and deliberately through the landscape, adding to the sense of foreboding and intrigue. The overall tone of the video is dark and moody, with a monochromatic color palette that enhances the sense of desolation and isolation. The fog and the rugged terrain contribute to a feeling of isolation and vulnerability, as if the figure is a lone entity in a vast, inhospitable world. The image evokes a sense of wonder and unease, inviting the viewer to speculate about the nature of the creature and the significance of its presence in this desolate landscape.

Empty prompt

1. busch marx pretend doomed moon clay scape video animation �stepped emulrubble clouds blurry forces 2. mars curiosity insight colossus suit frighrumble rumble godzilla 🎥: sdanimation nature blurry ufo devastation 3. sentinmonster re enigma unknown strange vfx forgotten footage enigma jst reminiscent astronauts wrecked dunes phenomenon 4. womensart filmmakers animation gothic taranvillain fingers syrian danielpadilla made": surface atop moon landed give

The scene depicts an alien-like creature standing on what appears to be the surface of another planet or barren landscape, possibly Mars given its reddish hue and rocky terrain. The environment is desolate with sparse vegetation visible as small protrusions from the ground. ### Detailed Description: 1. **Creature Appearance**: - The central figure resembles an extraterrestrial being. - It has multiple limbs that appear segmented like tentacles rather than traditional human legs. - Its body seems elongated but compacted into one form without distinct separation between head and torso. - The skin texture looks rough and bumpy, suggesting it might have some kind of armor plating beneath the outer layer for protection against harsh conditions. 2. **Environment Details**: - The sky above is hazy and overcast, giving off a gray tone which could indicate dust storms common on planets such as Mars due to low atmospheric pressure. - Rocks scattered across the foreground add depth while emphasizing how inhospitable this place may seem; they vary slightly in size yet maintain uniformity throughout their arrangement. 3. **Lighting & Atmosphere**: - Lighting suggests either dawn/dusk time frames where sunlight filters through clouds creating soft shadows around objects. - Overall atmosphere feels eerie because there's no sign life forms other than our protagonist here – adding mystery about whether this creature lives alone or if others exist elsewhere unseen by us at present moment captured within frame. 4. **Mood/Emotion**: - Given lackluster surroundings coupled with imposing presence of creature itself evokes feelings of isolation and perhaps even fearlessness towards unknown dangers lurking beyond view. 5. **Possible Interpretations**: - This image can evoke curiosity regarding existence possibilities outside Earth especially considering advancements made recently concerning space exploration missions aimed specifically toward discovering signs intelligent beings inhabiting distant worlds. In summary: A solitary alien stands amidst starkly unforgiving terrain under ominous skies hinting potential stories waiting untold behind each corner of universe unexplored thus far!

Prompt-based video editing

GPT-4o minimally rewrites the recovered inverse prompt according to the edit instruction; the edited prompt and only the first frame of the source video are then used for video generation, without access to the benchmark target prompt or any editing-specific training.

Edit instruction: Make the man and the boy suddenly fall asleep.

Source

MVIP Ours

RF-Edit [3]

Wan-Edit [4]

1 / 5

References

[1] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems (NeurIPS), 2023.

[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.

[3] Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing. Proceedings of the International Conference on Machine Learning (ICML), 2025.

[4] Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, and Mengyu Wang. FiVE-Bench: A fine-grained video editing benchmark for evaluating emerging diffusion and rectified flow models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.