-
Take it on yourself
You don't need
Melodramatic simplicity
You say God saves
And he knows best
I would wager he was saving his breathTake it on yourself
You don't need
Melodramatic simplicity
You say God saves
And he knows best
I would wager he was saving his breathLord have mercy let me paint the scene
Night was heavy like a rusted beam
The tower shifted underfoot
Kept not a secret from meLord have mercy could I get some sleep
Sun was tipping over Saturn's rings
Torching every monument
Crumbling cold at my feetWeighed down, weighed down
Weighed down, weighed downTake it on yourself
You don't need
Hyper-poetic dramatizations
You say the pen's mightier than the sword
Same breath: actions louder than words!Take it on yourself
You don't need
Melodramatic simplicity
You say God saves
And he knows best
I would wager he was saving his breathLord have mercy while I shake this dream
Light was sweeping every shadow clean
The vault of blue swung overhead
Drawing me out to the seaWeighed down, weighed down
Weighed down, weighed downTake it on yourself!
Take it on yourself!Farewell familiarity
Falling away from my wingsWeighed down, weighed down
Weighed down, weighed down
We asked an AI to make a video for “Weighed Down” and here is the result.
Back in July of 2022 our good friend, and mighty brother of one of our vocalists, William Huster asked if he might prompt an AI with one of our song’s lyrics. We sent him the lyrics and story to “Weighed Down” and he generated the video you see here. We called it our “Weighed Down Vid-zulAIzer”
William used a pairing of Disco Diffusion, Flowframes, and Adobe Premiere Pro to put it all together. Contrary to how most people imagine it, and have seen it work with apps like Lensa and Craiyon, there was actually quite a bit of narrative work, mathematical prompting, and final manual editing that went into this video.
We generated the frames with the support of Google Labs’ computational power. Due to the length of the wait time for each frame, we opted for a 10fps generation rate and then smoothed the final cut with Flowframes. The work also had to be done in parts due to limitations in active computational time. Each part was generated overnight, usually taking 6-8 hours, or around a minute per frame.
Outlined below is the code we used for each part of the video. These prompts indicate the content, style, and camera movement for the video. These are fun to look at if you want to geek out about it with us, but also useful if you decide you want to create something similar!
-
0:00 - 1:02 (Frame 0-620)
text_prompts = {
0: [
"A dark tower under a full moon on a stormy night:2",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
"people:-1"
],
110: [
"Lightning strikes from a dark cloud:2",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
"people:-1"
],
160: [
"Rain falling:2",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
"people:-1"
],
190: [
"A tower window:2",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
"people:-1"
220: [
"A young wizard sits at a desk writing in an old book",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
330: [
"A demonic choir encircles a young wizard",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
440: [
"A young wizard looks out of a tower window as candlelight flickers",
"acrylic painting",
"cool color scheme",
"text:-1"
],
490: [
"A rusted beam falls from a tower",
"acrylic painting",
"cool color scheme",
"text:-1"
],
550: [
"A spiral staircase going down to a secret door",
"acrylic painting",
"cool color scheme",
"text:-1"
],
}
MOVEMENT MAP
"translation_x": "0:(0),620:(0)",
"translation_y": "0:(0),439:(0),440:(-9),550:(0),620:(0)",
"translation_z": "0:(0),49:(0),50:(3),219:(3),220:(0),221:(-3),439:(-3),440:(0),620:(0)",
"rotation_3d_x": "0:(0),549:(0),550:(-3),619:(-3),620:(0)",
"rotation_3d_y": "0:(0),549:(0),550:(-3),619:(-3),620:(0)",
"rotation_3d_z": "0:(0),549:(0),550:(-3),619:(-3),620:(0)",
-
1:02-2:01 (Frame 0-590)
text_prompts = {
0: [
"A door opens into a stately bedroom with a prince’s canopy bed",
"acrylic painting",
"cool color scheme",
"text:-1"
],
50: [
“A sunrise over saturn",
"acrylic painting",
"cool color scheme",
"text:-1"
],
110: [
"Monuments towers and ruins in daybreak",
"acrylic painting",
"cool color scheme",
"text:-1"
],
160: [
"An athletic boy swims in the open sea",
"watercolor",
"text:-1”
],
210: [
"A dark tower under a full moon on a stormy night",
"Magazine, collage, photograph, Kolaj Magazine",
"Text:-1"
],
250: [
"Lightning strikes from a dark cloud",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
305: [
"A young wizard sits at a desk writing in an old book",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
360: [
"Dark and creepy silhouettes in doorways",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
415: [
"Pages of a book flying around in a storm",
"Magazine, collage, photograph, Kolaj Magazine",
"text:-1"
],
480: [
"A spiral staircase going down to an open door",
"acrylic painting",
"cool color scheme",
"text:-1"
],
540: [
"A sunrise over a calm ocean",
"acrylic painting",
"cool color scheme",
"text:-1"
],
}
MOVEMENT MAP
"translation_x": "0:(0),590:(0)",
"translation_y": "0:(0),590:(0)",
"translation_z": "0:(3),249:(3),250:(0),539:(0),540:(3),590:(3)",
"rotation_3d_x": "0:(0),479:(0),480:(-3),539:(-3),540:(0)",
"rotation_3d_y": "0:(0),249:(0),250:(-1),479:(-1),480:(-3),539:(-3),540:(0)",
"rotation_3d_z": "0:(0),479:(0),480:(-3),539:(-3),540:(0)",
-
2:01-2:32 (Frame 0-310)
text_prompts = {
0: [
"A small raft with a boy on it",
"watercolor",
"cool color scheme",
"text:-1"
],
60: [
“A big sun in a bright blue sky over the ocean",
"watercolor",
"cool color scheme",
"text:-1"
],
115: [
"A boat raising its sails",
"watercolor",
"cool color scheme",
"text:-1"
],
160: [
"Anchors falling from the sky",
"Magazine, collage, photograph, Kolaj Magazine”
"text:-1”
],
260: [
“Old wooden tower doors",
"Magazine, collage, photograph, Kolaj Magazine",
"Text:-1"
],
}
MOVEMENT MAP
"translation_x": "0:(0),310:(0)",
"translation_y": "0:(0),310:(0)",
"translation_z": "0:(3),259:(3),160:(0),259:(0),260:(3),310:(3)",
"rotation_3d_x": "0:(0),159:(0),160:(3),209:(0),210:(-3),260:(0),310:(0)",
"rotation_3d_y": "0:(0),310:(0)",
"rotation_3d_z": "0:(0),310:(0)",
-
2:32-3:01 (Frame 0-290)
text_prompts = {
0: [
"A large shattered stained glass window",
"Magazine, collage, photograph, Kolaj Magazine”
"text:-1”
],
110: [
“A sunrise over a calm ocean",
"acrylic painting",
"cool color scheme",
"text:-1"
],
160: [
"A large albatross flying",
"acrylic painting",
"cool color scheme",
"text:-1"
],
220: [
“A small raft with a boy on it”,
"acrylic painting",
"cool color scheme",
"text:-1"
],
}
MOVEMENT MAP
"translation_x": "0:(0),290:(0)",
"translation_y": "0:(0),290:(0)",
"translation_z": "0:(3),229:(3),230:(0),290:(0)”
"rotation_3d_x": "0:(0),290:(0)",
"rotation_3d_y": "0:(0),290:(0)",
"rotation_3d_z": "0:(0),290:(0)",
Weighed Down
~
Weighed Down ~
“TMW the AI absolutely nails the lighting strike”