Virtual Marionette

Digi Mario – A digital rod puppet style with physics. A hand manipulated puppet with leap motion controller. One hand controls the puppet and the other simulates the puppeteer virtual hand for physical interaction.

[vimeo 110798138 w=580&h=326]

This video shows how to animate a puppet with just one hand using physics to recreate the marionette aesthetics.

Marionette animation is fascinating but requires a lot of skills, this virtual marionette is much simpler but the digital puppeteer can control many aspects of the puppet. With few training the puppeteer can bring this puppet to life in a traditional style.
All fingers are mapped to the face controls and the hand is mapped to the head. When you move the head, the body will follow along like if we had a rod connected to the head.
Interesting effect is when you turn your hand using your pinky finger first, the eyes will look to the target and the head will follow the eyes.

framework: Leapmotion v2+ Unity

[vimeo 110452298 w=580&h=326]

Demonstration of full hand control for expressive digital puppetry

This video shows how to animate a digital puppet in real-time with just one hand in an expressive manner.
It is part of a PhD research in the digital puppetry field.

Stringless hand controller (a metaphor for the marionette controller)
Hand + 5 finger controls different aspects of the puppet

Hand: position and orientation of the puppet
Pinky finger: Eye Pupils rotation in all directions
Index finger: Eyelashes rotation in the +Y and -Y axis (open and close)
Middle finger: Right Eyebrow blend shape deformation for character expressions
Ring finger: Left Eyebrow blend shape deformation for character expressions
Thumb: Mouth (open and close) blend shape deformation

There are different degrees of freedom (DOF) for each finger mapped to a certain puppet control.
The Hand as 6 DOF (position and rotation), you can move your hand freely around the tracking area but must be carful with the occlusion problem;
The pinky finger has 3 DOF for rotation of the eye pupils, and although controlling the pinky finger independently as some constraints, for this kind of small motion is more then adequate.
The middle and ring fingers are have more constraints and it is very hard to controlled them independently, so I mapped just 1 DOF (up and down) of each finger to the eyebrows.
The index finger as more potential because you can control more degrees of freedom, I tried using the index to control the eye pupils but the results were not so great. Instead, the index finger is mapped to the eyelashes rotation with just 1 DOF (up and down)

This is a good direction for digital puppeteers that have full control of the character expressivity. It require some training to act like a character, but that´s the magic of puppeteering.

This prototype can animate two different full controlled puppets for interaction.
A powerful model for performance animation using digital puppeteering techniques.

Hardware: Leap motion device for hand tracking and a macbook
Software: Unity Engine

The Porto International Marionette Festival (FIMP) is out there again showing the diversity of puppetry from all the world.

Although the puppetry plays are quite interesting I´m delighted with the documentary film cycle called the Marionnettes du Monde from Acrobates Films where I had the opportunity to learn more about puppetry in Taiwan and Poland.

Marcelo Lafantona won the award for best male performance with Prometeu at the 16th International Puppetry Festival in Varna – Bulgaria.

The renewed Prometeu system was performed with great success by the hands of Marcelo. A great performance presented to the Bulgarian audience. The Juri although not knowing the spoken language as chosen Marcelo´s performance as the best male performance.

The new system is more stable and flexible helping Marcelo to be more focus in his performance. Marcelo is the one man show, controlling lights, sounds, video and animation and all the puppeteering. And although it was very complex to create all the stuff is changed by the puppeteer in real-time with just one button.

A very important recognition from one of the oldest Eastern Europe festivals with a great tradition in the art of puppetry.

Congratulations to Marcelo Lafontana.

Photos from TMPC – Varna, Stage Filial – Drama Theater Stoyan Bachvarov

Visting Varna in Bulgaria to the 16th International Puppetry Festival.

Lafontana Formas Animadas was invited to present the Prometeu shadow play at this festival. One of the oldest Eastern Europe puppetry festivals with 45 years old.

We built a new compact version of Prometeu which travels inside a flat case  developed to go around the world. It was our first challenge and it passed with distinguish, Marcelo Lafontana won the award for the best performance. We were very happy to present our show at Varna.

From the 1th Octobre to 7th October I have had the privilege to watch 11 great performances from puppetry companies from all over the world.

From string to direct manipulation, from the most classical puppetry to the most contemporary approaches the Varna festival presented a wide range of puppetry aesthetics  and puppetry methods for all audiences.

I would like to congratulate the organization for there professionalism handling with all cultural differences representing 10 different countries.

Here is a list of all the puppetry shows that I was able to see,  all of them contributed in some way to opening my eyes to this wonderful theatrical art.

La Mar de Marionetas – Madrid, Spain
SPEAK, YOU´RE NOT ALONE

Sofia Puppet Theatre – Sofia
TALE OF THE WANDERING KING

Batumi Puppet and Youth State Theatre – Batumi, Georgia
COLORED FOG’S SEASHORE

State Puppet Theatre – Varna
LITTLE WIZARDS

Harpe en Perigord & La Mar de marionetas – French-Spanish co-production
SHAKESPEARE’S DAUGHTER TELLS HER MIDSUMMER NIGHT’S DREAM

Puppet Theatre Haet-Sal – Suwon, Korea
HEUNG-BU JEON

Baruti Teatro – Athens, Greece
MOMENTOS

Sofia Puppet Theatre – Sofia
THE TEMPEST

State Puppet Theatre – Plovdiv
THE LONG JOURNEY OF THE LITTLE CROCODILE

State Puppet Theatre – Varna
SILENT LEGENDS

The Key Theatre – Tel Aviv, Israel
WHEN ALL WAS GREEN

Jim Henson Red Book

A sort of a diary of Jim Henson activity. It is an important Online reference.

Prometeu – Real-time Illusion

Luís Leite and Marcelo Lafontane received an honorable mention at Avanca Festival 2014.

International Conference on Film – Art, Technology and Communication.

The prize Eng. Fernando Gonçalves Lavrador was ax-aequo to the Finn researcher Jouko Aaltonen from Aalto University and to the Brazilian Carlos Júnior Rosa from São Paulo University.

There were two honorable mentions, one to the portuguese researchers Luís Leite and Marcelo Lafontane and the other to the Danish Michael Morgan from the European Film College.

The internationally jury for this prize was composed by academic

International academic Juri commission : Francisco Garcia (Spain), Wai Luk Lo (Hong Kong), Yen-Jung Chang (Taiwan) and by the portuguese Anabela Oliveira, José Ribeiro, Rosa Oliveira, Pedro Bessa and José Marta.

In 24th of July of 2014 I will be presenting the paper Prometeu – Ilusão em tempo real at the Avanca Festival in Portugal.

Leite, L., Redondo, M. (2014) Prometeu, Ilusão em tempo-real. Conferência Internacional de Cinema, Arte e Tecnologia. Avanca.

Prometeu is a multimedia theater play inspired by the Indonesian shadow puppetry, Wayang Kulit. This multidisciplinary project narrows the boundaries between theater and film, past and present, reality and illusion. Merging different art forms brings new experiences to the artists and to the public; in particular the amalgamation of puppetry with film and other disciplines contributes to a new concept, what we can call the performance cinema or ” Live Cinema”.

Using multimedia technologies and cinematic language this project motivates a different paradigm from the traditional reading of the narrative sequence. It promotes the discussion about the boundaries of theater and film, and the role of actors, puppeteers, animators and directors in this particular art form.

Preserving the heritage of traditional shadow theater, Prometeu overcomes the typical physical constraints found in traditional puppet theatre through film and multimedia techniques. Real-time post-production, performance animation or interaction techniques provide a new mean of creation to artists.  By using sensors the artists can expand the interaction. By using live cameras the performer becomes the director creating the sequence in real-time. Techniques such as “luma-key”, procedural animation or gesture tracking enhance the live experience making the performer into a sort of magician.

In Prometeu one puppeteer, performs with virtual and real objects and controls all the show remotely.

In this paper we present the methodology and the implementation of our technology as well as the aesthetics and conceptual aspects. 

LeapMuppet is a digital hand puppet based on the Leap Motion interface.
It was developed with the new skeletal tracking in Unity.

An inspiration on the traditional glove puppet, in particular the muppets, from the great Jim Henson.
This is a digital puppetry proof of concept. The ideia is to bring the art of traditional puppetry manipulation to performance animation.
Not real´istic animation, but believable animation…
Do not believe that the puppet is for real, just believe that the puppet is alive.

A simple character made in Maya, which is real-time animated with the motion of the hand.
I use a skeleton to drive the nose and the head of the puppet (position, orientation), and blend shapes to control the mouth and the eyebrows.
The palm of the hand drives the head (direction free); the index finger drives the nose (direction free); the thumb controls the mouth (up and down); the ring finger controls the left eyebrows (shrink / expand); the pinky finger controls the right eyebrows.

With just one hand you can give great expression to the puppet. Imagine what you can do with two hands ?

Let´s have some fun with virtual puppets.

TheToyWorlds was evaluated at a junior school as part of the virtual marionette research.

Picture 1 – Childrens interacting with the game at the classroom.

Introduction

We challenged children under the ages of 5 and 6 to grab a virtual puppet and play with it.

The main goal was to better understand how children under these ages who never had contact with digital interfaces based on body motion respond to the interaction. The ToyWorlds is a two player game where the player controls the virtual marionette with their hands. The left hand manipulates the body of the puppet and the head follows the right hand motion. The players grab the puppet with the left hand and move the target where the puppet is looking with the right hand. In this way we challenge the participants to control their own body as if it was a marionette controller. This experience is somehow near to how puppeteers manipulate with their puppets, using rods, strings, or other kind of controller.

Picture 2 – Simulation of the ToyWorlds puppet manipulation using the two hands

In puppetry the head of the puppet is essencial to transmit emotions and to help the audience to understand the direction of the action or motion. By fowling the players right hand with their heads, the virtual puppets call the attention of the audience and a contribution to a more expressive animation. Although this is a very simple way to control virtual puppets by just using our two hands it is a very challenging interaction because our hands are not visually represented in the scene. In this way, we must calculate the position of our hands by the position of the puppet´s body and head target. And because the result does not mirror our body exact motion, we need the capacity to abstract from the real motion to the represented motion. This indirect manipulation creates a distance between the player and the puppet witch we want to understand.  This is one important  aspect we want to explore, study and measure.

picture 3 – The right hand makes the puppet to rotate. The puppets head follows the players right hand forcing the body to rotate.

By assigning two different tasks to our two hands we challenge the players to take control of their bodies, an abstraction of the body which may be very difficult to childrens under 6 years old.

The virtual marionettes are rag-doll puppets, dynamically affected by gravity and collisions, the result is a very natural and rich animation.

picture 4 – screenshot of the game

 From inside out, the game narrative!

In Marta´s bedroom, there are strange things happening when she goes to sleep. She dreams with paper boxes that fill her room creating a big mess. Because she is a little lazy to put her bedroom in order, she created two little imaginary friends that help her to clean up. Joaquin wears a green shirt and his the uncle of António which wears a orange shirt, both are paper figures and very hard workers. They can only grab the boxes when they change their colors, Joaquim grabs the green boxes and António grabs the orange boxes. When the boxes change to the blue color, both puppets can pick them by just looking at them using their respective looking target square. Green and orange boxes are grabbed with the bodies of the puppets using the players left hand and the blue boxes are picked up with the looking target using the players right hand. After grabbing the boxes the player receives a score of one point to the green or orange boxes and ten points to the  blue boxes. The boxes disappear if grabbed or return to their natural color after a certain time.

The players give life to this two little figures and have one minute and an half to grab the boxes. They start by making the calibration pose and the first to get their pose calibrated takes control of the first player witch is Joaquim.

This game evaluates body motor control, an important aspect for the puppetry manipulation.

In the table above we show the game results of this experiment. Two of the 22 children were not able to grab any color box because they felt stressed by the competition.

ToyPaper Game Junior challenge