CTRL-Adapter: An Efficient and Versatile Framework
for Adapting Diverse Controls to Any Diffusion Model

Generated Examples

We show examples from both U-Net based models (I2V-GenXL & SDXL), and DiT based models (Latte & Pixart-α)



Video Generation with Condition Control (w/ I2V-GenXL; U-Net based)


"A fish swimming"
+
Control Generated Video
"A man riding a motorcycle jumps off a mountain"
+
Control Generated Video
"Close-up of a majestic white dragon
with pearlescent, silver-edged scales,
icy blue eyes, elegantivory horns, and
misty breath. Focus on detailed facial
features and textured scales, set
against a softly blurred background"
+
Control Generated Video





"A car flies over a hill"
+
Control Generated Video
"A white and orange tabby cat
is seen happily darting through a
dense garden, as if chasing something.
Its eyes are wide and happy
as it jogs forward, scanning the
branches, flowers, and leaves as it
walks. The path is narrow as
it makes its way between all
the plants."
+
Control Generated Video








"A bird flying over a forest."
+
Control Generated Video
"A miniature Christmas village with
snow-covered houses, glowing
windows, decorated trees, festive
snowmen, and tiny figurines in a
quaint, holiday-themed diorama
evoking a cozy, celebratory winter atmosphere"
+
Control Generated Video



"A woman wearing blue jeans and a
white t-shirt taking a pleasant stroll in
Mumbai India during a beautiful sunset"
+
Control Generated Video


Video Generation with Condition Control (w/ Latte; DiT based)


"A 2d abstract japanese animation where drops of ink in water form into lifelike creatures that swim and interact with each other, creating an ethereal underwater world made entirely of flowing, merging colors"
Control Generated Video

"A giant, towering cloud in the shape of a man looms over the earth. The cloud man shoots lighting bolts down to the earth"
Control Generated Video

"A medium sized friendly looking dog walks through an industrial parking lot. The environment is foggy and cloudy. Shot on 35mm film, vivid colors."
Control Generated Video


Video Generation with Multiple Control Conditions

"A small child and an adult standing
in shallow ocean waters along the beach"
+
Controls Generated Video





"A man dancing"
+
Controls Generated Video




"A woman wearing purple overalls and cowboy boots
taking a pleasant stroll in Johannesburg South Africa
during a beautiful sunset"
+
Controls Generated Video




"A skateboarder mid-trick, airborne above
a bench, wears a casual outfit and a beanie,
displaying focus and athletic skill"
+
Controls Generated Video


Video Generation with Multiple Control Conditions (using Patch-Level MLP)

"butterfly"
First Frame + Controls Generated Video


"dog agility"
First Frame + Controls Generated Video


"flamingo"
First Frame + Controls Generated Video


"snowboard"
First Frame + Controls Generated Video


"tennis"
First Frame + Controls Generated Video




Video Editing via Combining Image and Video Ctrl-Adapters

(1) Control Condition Extraction

Input Prompt (2) Generated Frame
(Generated by SDXL + Ctrl-Adapter)		 (3) Generated Video
(Generated by I2VGen-XL + Ctrl-Adapter)

A camel with rainbow fur walking.

A zebra stripped camel walking.

A camel walking, ink sketch style.

A camel walking, van gogh-style.






Text-Guided Motion Control

Initial Frame Object Masking Input Prompt Generated Video
(Generated by I2VGen-XL + Ctrl-Adapter)

A white and orange tabby alley cat is seen darting across a back street alley in a heavy rain, looking for shelter.

A white and orange tabby cat is darting through a dense garden, as if chasing something

An elk with impressive antlers grazing on the snow-covered ground







Video Style Transfer

Initial Frame Shuffled Input Prompt Generated Video
(Generated by I2VGen-XL + Ctrl-Adapter)

A miniature Christmas village with snow-covered houses, glowing windows, decorated trees, festive snowmen, and tiny figurines in a quaint, holiday-themed diorama evoking a cozy, celebratory winter atmosphere

Stop motion of a colorful paper flower blooming

Beautiful, snowy Tokyo city is bustling







Video Generation with Sparse Frames as Control Condition

"Fly through tour of a museum with many paintings
and sculptures and beautiful works of art in all styles"
+
Sparse Inputs
(Condition is given for 4 out of 16 frames)		 Generated Video
 ... 


"Reflections in the window of a train traveling through the Tokyo suburbs."
+
Sparse Inputs
(Condition is given for 4 out of 16 frames)		 Generated Video
 ... 


Zero-Shot Generalization on Unseen Conditions

"An old man wearing purple overalls and
cowboy boots taking a pleasant stroll in
Mumbai India during a beautiful sunset"
+
Condition Controls Generated Video

Training: Depth Map

Inference: Normal Map




"An extreme close-up of an gray-haired man with a beard in his 60s, he is deep in thought
pondering the history of the universe. He sits at a cafe in Paris, his eyes focus on people offscreen. As they walk, he sits mostly motionless, he is dressed in a wool coat suit coat.
With a button-down shirt, he wears a brown beret and glasses."
+
Condition Controls Generated Video

Training: Depth Map

Inference: Line art




"This close-up shot of a chameleon showcases its striking color changing capabilities.
The background is blurred, drawing attention to the animal's striking appearance.
The chameleon's vibrant colors and unique texture are the focus of this shot."
+
Condition Controls Generated Video

Training: Depth Map

Inference: Softedge


Image Generation with Condition Control (w/ SDXL; U-Net based)

Prompt Control Generated Image

Cute fluffy corgi dog in the city in anime style

happy Hulk standing in a beautiful field of flowers, colorful flowers everywhere, perfect lighting, leica summicron 35mm f2.0, Kodak Portra 400, film grain

Astronaut walking on water

a cute mouse pilot wearing aviator goggles, unreal engine render, 8k

Cute lady frog in dress and crown dressed in gown in cinematic environment

A cute sheep with rainbow fur, photo

Cute and super adorable mouse in black and red chef coat and chef hat, holding a steaming entree

a cute, happy hedgehog taking a bite from a piece of watermelon, eyes closed, cute ink sketch style illustration






Image Generation with Condition Control (w/ Pixart-α; DiT based)

Prompt Control Generated Image

A plate of cheesecake, pink flowers everywhere, cinematic lighting, food photography

Darth Vader in a beautiful field of flowers, colorful flowers everywhere, perfect lighting

A micro-tiny clay pot full of dirt with a beautiful daisy planted in it, shining in the autumn sun

A raccoon family having a nice meal, life-like