TL;DR
A PyTorch fork of neural-dream adds RAFT optical flow and occlusion masking to apply DeepDream to video with reduced flicker and smoother temporal transitions. The repo includes a new video_dream.py CLI, demo outputs, dependency and model download instructions, and guidance for lowering per-frame iterations.
What happened
A GitHub fork of the neural-dream PyTorch implementation extends DeepDream from single images to video by introducing temporal consistency features. The project uses RAFT optical flow to warp previously generated ‘dream’ frames into the next frame, allowing hallucinations to track objects across frames instead of restarting per-frame and producing flicker. An occlusion masking step detects when objects move in front of one another to reduce ghosting artifacts. The repository supplies a dedicated CLI script (video_dream.py) that accepts both video-specific flags — such as blend ratio, independent (disable temporal consistency), update interval, and temp directory options — and the standard image dreamer parameters (layers, octaves, iterations, etc.). Demo videos are provided for comparison between temporally-consistent and independently processed frames. Installation requires common packages (PyTorch, torchvision, OpenCV, NumPy, Pillow) and a model-download script to fetch compatible Inception/GoogLeNet weights.
Why it matters
- Brings temporal consistency to DeepDream video outputs, reducing frame-to-frame flicker common in independent-frame processing.
- Optical flow warping enables hallucinations to follow moving objects, producing more coherent animations.
- Occlusion masking helps avoid artifacts when scene geometry changes cause overlap between objects.
- The integration reduces the need for many optimization iterations per frame, potentially lowering compute per-frame.
Key facts
- This project is a fork of neural-dream implemented in PyTorch.
- Video processing uses RAFT optical flow to warp previous dream frames into the current frame.
- Occlusion masking is applied to detect object overlaps and prevent ghosting artifacts.
- A new CLI script video_dream.py accepts both video-specific and single-image DeepDream arguments.
- Recommended setting for video: -num_iterations 1, since temporal consistency builds on prior frames.
- Video-specific options include -blend (mix ratio), -independent (disable flow), -update_interval, -temp_dir, and -keep_temp.
- Dependencies listed: PyTorch, torchvision, OpenCV, NumPy, Pillow; installable via pip -r requirements.txt.
- A download script is provided to fetch Inception/GoogLeNet model files (python models/download_models.py).
- Default memory usage for standard execution is around 1.3 GB of GPU memory (per the repository guidance).
- Multi-GPU execution is supported via -gpu and -multidevice_strategy flags (layer splitting examples provided).
What to watch next
- Video processing is computationally expensive; expect slow runs because each frame runs DeepDream plus optical flow computation.
- The -update_interval option updates the output file every N frames so you can preview progress while processing.
- Real-time processing feasibility — not confirmed in the source.
Quick glossary
- DeepDream: A technique that uses neural network activations to iteratively amplify patterns in an image, producing hallucinatory visuals.
- Optical Flow: A method for estimating apparent motion between consecutive frames to map how pixels move over time.
- Occlusion Masking: A technique to detect regions where objects are covered or uncover one another, used to avoid visual artifacts when compositing frames.
- RAFT: A specific optical flow algorithm model used to produce dense flow fields between video frames.
- Temporal Consistency: A property of video processing where transformations are coherent across frames, reducing flicker and discontinuities.
Reader FAQ
How do I run DeepDream on a video with this project?
Use the video_dream.py script, for example: python video_dream.py -content_video input.mp4 -output_video output.mp4 -num_iterations 1.
Which dependencies are required?
The repository lists PyTorch, torchvision, OpenCV, NumPy, and Pillow; install with pip install -r requirements.txt.
I get out-of-memory (OOM) errors — what can I do?
The README suggests reducing -image_size, using the -backend cudnn on GPU, or lowering resolution for video inputs.
Can this run on Apple Silicon?
The source mentions experimental MPS support for Apple Silicon in the neural-dream codebase; for specifics, consult the repository files.
Is real-time processing supported?
not confirmed in the source
deepdream-video-pytorch This is a fork of neural-dream, a PyTorch implementation of DeepDream. This fork introduces optical flow estimation and occlusion masking to apply DeepDream to videos with temporal consistency. Features…
Sources
- Show HN: DeepDream for Video with Temporal Consistency
- deepdream-video-pytorch
- Controlled Temporally-Consistent DeepDream on Videos
Related posts
- Google Adds AI Inbox Beta to Gmail That Summarizes Emails, Suggests To‑Dos
- Google unveils AI Inbox that replaces Gmail’s message list with to-dos and summaries
- UK regulators press X after Grok reportedly produced sexualised and child images