How Runway took The Late Show edits from five hours to five minutes
Hi! Thanks so much for joining us today. To start, can you tell us a bit about your Late Show team and how you fit into the larger production.
Andro: We’re a team of six people who handle all the graphics on The Late Show with Stephen Colbert. We do the animation, illustration, and static graphics. We have a separate post department that does the editing.
Colbert is one of the most graphics-heavy late night shows on TV. What is the philosophy behind using graphics?
Andro: Stephen always mentions having a toolbox, like this is another comedy tool in the toolbox. Often it’s more video-based jokes, so putting words into people's mouths and editing things together. He definitely reaches for different tools, depending on what's happening at the moment, and what viral video is out there.
Does your team keep track of what’s happening in the world of viral videos?
Andro: There’s a Research Department, and the writers are all on top of current events. Sometimes, we'll listen in on the writers’ pitch meetings just to hear what the direction is going in. But it really all starts with them.
From there, it goes through the production pipeline to us, where there's back and forth communication about how to execute it and what the timeline is. And then we just crank it out. They put a lot of faith in us to just make it funny.
Ryan: Yeah, I think as a graphic artist on a comedy show, you just have to have a good visual vocabulary. You have to be familiar with all the old movies, TV, and music that you can pull from and reference from a graphical standpoint.
You’ve both been working in late show VFX for many years. How did you find Runway? What problem were you trying to solve?
Ryan: I found it by watching a rotoscoping competition on YouTube. It was four guys from Corridor Crew, and they were all using different methods of rotoscoping to see who could edit the video, the fastest and the most accurate.
One of them was like, there's this thing and it just does it in like a click. It blew my mind because here I am in After Effects, with the Rotobrush doing it frame-by-frame myself. And this guy was done in three minutes on a really complex shot.
I started using it in secret. And then after a while, I was like, Andro, we gotta get on this now.
Andro: Ryan mentioned it to me, and I was like, this is a no-brainer. It’s just incredibly impressive how fast and accurate it is.
That’s great to hear. Can you tell us about a project where Runway made a big difference?
Andro: We used Runway for a music video commemorating the anniversary of the Americone Dream ice cream by Stephen Colbert. They put together a music video with Sara Bareilles and Josh Groban. The writers wrote the lyrics, and there was talent onboard, a script written, and a shooting schedule done.
They made a giant prop costume for a dancer to wear, and the idea was that throughout the song, they were exaggerating the ice cream’s impact on society at large. So we were putting the Americone Dream in various places throughout history.
There were all kinds of ideas, and it was mostly putting them into existing footage. So, they shot it, but the issue was there’s a lot of colors in the paint. So if we shoot on green, there’s too much green in the paint. It meant a lot of hand masking or matting. There was too much in the ice cream as well. So the art team in the studio built a magenta backdrop, thinking that we could try to key it, which was kind of successful, but not really.
We went to Runway in order to pull all the mattes for it.
I think if we hadn’t had Runway, it would have been a disaster, and a much bigger lift. You could kind of pull a key, but there were a lot of rough edges and a lot of stuff was disappearing. The Runway matting process was perfect and it allowed us to actually focus on shadowing reflections.
Ryan: Yeah, we tracked it and then we needed floor reflections. I had the idea of putting plexiglass underneath the paint so that it could get accurate reflections. And then, we just dropped it into the shot and I did a little bit more roto work because he has to be behind the dancers and Katy Perry and the sharks, and everything.
It sounds like you’re on very quick turnaround schedules. How long, on average, would you say you have for a project?
Andro: Turnaround varies, but for certain projects it's a few hours. We’ll find out about something around noon, and we’ll have to show it at rehearsal, at three. For the ice cream clip, there was more of an organized shoot. We spent one day on a Zoom, because people had to be separated and whatnot. So we were on Zoom, kind of peering in on the shoot and weighing in on the angles and lighting and whatnot.
And then I think we had one day to do the composites.
Ryan: Yeah, that was a pretty ample amount of time. We felt that we had time to polish it, as opposed to just getting it out the door.
But by doing those fast rotos in just a few minutes on normal days where we only have a couple of hours to make a bunch of different effects shots, Runway is extremely helpful and lets us focus on the other parts of the comping, or animation or other creative things we have to do. It really speeds up our pipeline.
Looking back on this process, how much time do you think Runway saved you?
Ryan: Oh boy. I mean, I’ve done a shot in five minutes, which normally would have taken me five hours. You're not going through it frame by frame, you're just tapping a couple of things and it does it. It’s got a very barebones interface, because you don't really need to tweak it.
It’s the kind of thing that anybody could jump into, even without knowing a lot of compositing software.
Andro: It’s definitely hours, like multiple ten hundred times the original amount of time, is saved. It's pretty incredible for people just breaking into VFX. Rotoscoping is the most tedious thing. And if it doesn't look good, it kills the whole project. With Runway, you can focus on those other things knowing that the roto is going to happen, it's going to be clean.
Ryan: I think it's also a bit of a Pandora's box situation because now that it's clear what we can do in the time that we're allotted, like, it makes me nervous with what we might be asked to do next.
It’s clear from this clip that your work in the editing department plays into the overall comedic effect. On the whole, how do you think VFX can amplify or enforce comedic storylines?
Andro: That’s an interesting question. I think sometimes it’s about taking something that’s clearly fake, and making it look real. Though that’s a dangerous thing in this world. But sometimes you make something that looks real, and that could be the comedy. So the joke is that it looks real.
The point in that clip is that the ice cream is sneaking up behind Clint Eastwood when he was on stage at the Oscars.
The realism of it from beginning to end, knowing what angle to shoot it at and getting a proper map for the for the roto and then the proper reflections and keys, it's not technically a joke or comedy, but the fact that it looks like it was really there and it really happened, can play into the joke.
Ryan, what do you think?
Ryan: Yeah, I agree it's that juxtaposition of the absurd. Appearing normal, or normalized, can elevate it to a comedic level. This is a case where we are given assets to use, and it's obvious where they need to go. There’s not a lot of creative choices necessarily that we have to make, as opposed to a project where they're like, animate something from scratch, make something from scratch. With this, there’s a whole coordinated shoot, there's a script and we have to do the best job that we can with what they give us.
Are there other projects you can see yourself using Runway on?
Andro: We do a lot of character animations, cartoon-type animations and taking a person and putting them in a movie, saying a certain line from a movie. There's Runway software that lets you insert those words into that person's mouth, which is kind of frightening, to think that you could just make someone say something else. But it's very interesting. We're just starting to explore it now.
Ryan: I did a quick test last week, where I used Biden from a press conference and made him say something. And then I used Runway's roto to cut his head out and then tracked it back on to footage of Biden. And I was kind of surprised at how easy it was.
Now that you’ve had experience with Green Screen, are there other parts of the production process that you think Runway could simplify?
Andro: Well, we spent a lot of time tracking footage, which is usually the first thing that you have to do. Set it up. And then, you can start being creative. Sometimes it works. Sometimes it doesn't. Sometimes it's a little janky. Using A.I. to expedite that tracking process would be extremely helpful.
Ryan: Yeah, definitely. I also understand there was something they had for depth of a scene. So you're looking at a shot, and it could understand what the space of that shot is, and then you could put things in the middle ground or the background of something and then use the roto tool to put things back on top of it.
I've done things with the Rotobrush and drawing spines, and tracking mattes and you have like 100 mattes to cut a person out. With this program, you click a couple of buttons, and it was finding everything. And even with very fast, moving footage, it came awfully close to being perfect on the first shot. I saw it and I was like, "this is witchcraft!" It’s hard to wrap your head around it. What computers do this? It's pretty, amazing.