Using AI in VFX respectfully, by Joshua Eiffel

Wild Island were asked to handle the VFX, Grade and Finishing on an exciting campaign for Herman Miller. With the entire office covered in their chairs and being a design geek myself, this was a really special opportunity to work with the legendary company.

Joshua Eiffel, VFX Artist & Compositor

Out of the many shots I worked on as VFX lead, one presented a unique challenge involving AI which I wanted to cover here today. As the deadline was approaching, Herman Miller felt a shot displaying an array of their classic chairs with colourful individuals was feeling too empty at the top. The initial belief amongst production was that it may be solved by copying and pasting some of the individuals from the front to the back. However, that idea, whilst good on paper, contained some tricky visual walls.

Lighting — although a clean looking shot.  The lighting changes dramatically across it. Some areas are dark with soft shadows, and others well lit with hard ones.

Occlusion — quite simply there’s people in the way of what we need!  If we can’t see what we need, we can’t reuse it. This limits what we can sample for our final shot.

Attack of the clones — copying the same people, looking the same, doing the same thing tends not to go unnoticed by our clever eyes. This again was a really special campaign with a great feeling of production value, we really didn’t want to introduce a cheap looking Achilles heel here by literally copying and pasting.

It was going to be a lot of work to isolate the complex crowd and paint in the missing parts… was it all going to be worth the time, money and effort in order to simply clone what was already there?  It felt like we weren’t really adding enough value with the time spent. What if there was a fourth option?…

Generative AI and respecting what is real

Now, lets be honest, AI is not a magic bullet, or a magic button.  And for good reason.  A huge amount of intent and curation  that has gone into the Art Direction and styling during the shoot, and that it needed to be respected as much as possible.  But this was also a bit of an impossible situation – how else can we introduce brand new extras without reshooting or using resource heavy CG with the deadline looming? 

Here are some of the steps and considerations that we took to try and pull it off in a way that felt respectful the the production team and the high quality of the advert:

First of all, we collected several references from the shoot to highlight the underlying style.  We trained a local AI model on this so that the work could not be copied by users elsewhere.   Once this was done, we began generating a huge amount of images, constantly changing parameters in the presence of the Art Director  until we got 5 models that felt right in that world that they had been building.

The chairs! The chairs were most important.  We mainly used Photoshop to recreate existing ones from new angles.  But where needed we generated slightly mixed features to create some variety.

Once we had our short list of characters and chairs, we switched to the traditional method of using Photoshop to composite our selected characters into position. This would form the basis of our video input.  Finally, we would use upscaling tools to increase the resolution.

Now we needed to animate the characters using video models.  Again, everything was generated on our local machines to not break NDAS and keep the work strictly confidential.  It takes a bit longer, but it means that we have full control over how the advert is being distributed.

Quality control — we needed to grade our results in the same way as everything else to keep the commercials visual consistency.  For this reason we worked with the LOG images.  One area of difficulty here however is that most Video AI is still producing lo-res 8 bit images.  This is being improved all the time (we now often output our AI results in 16 bit colour), however it’s still very normal to run into image fidelity issues when grading, as well as artifacts and deformities. We have to keep this in mind when it comes to quality control. Often, it simply means returning to traditional Nuke or Flame retouching and clean up.

Finally, with a lot of composed references, generations, and a lot of traditional 2D compositing in Nuke, we got to a place we were happy with.

Some takeaways…

Keeping things fully collaborative

I really can’t imagine this experience without having creatives from Herman Miller and Look in the room with us.  It was so important to have their feedback and sign off on what was being generated to steer things in the right direction visually.

Likewise, none of the AI generations could have been achieved without directly referencing and building upon the work already achieved by the art direction and styling that was captured in the shoot.

Working with AI is far from just using prompts

A lot of what was achieved was by preparing pre-vis for the AI models, receiving it, polishing it, customising it, and sending it back for another iteration.  It was more like a conversation of the human and AI compositor than just a one off request.  We often also need to create “chains” of generations to address different output issues.  For example, the style and art direction often is better controlled by an image model.  We then feed that to a video model which may have great animation yet poor resolution, leading us  to use an upscaler AI model to improve that result afterwards.

Is AI “there yet”?

Recently someone compared this question to asking it in 1993 about CGI, way before the modern tools we have now were available.  The answer is that it’s difficult, it’s broken into many pieces that are yet to grow, many tools are toy like in their fidelity, but just like in Jurassic Park in 1993, by mixing old and new you can do some pretty amazing feats.  Here, just by using one computer, we were able to solve a situation that would have needed a reshoot previously to achieve in this way.

So it’s very exciting to see what tool will be here tomorrow.