Virtual Marionette

## Digital Hand Puppetry

ManiPullAction is a digital hand puppetry prototype built for a research project called Virtual Marionette. It was developed in March 2015 to evaluate the hand dexterity with the Leapmotion device to manipulate digital puppets in real-time. One year after its development maniPULLaction prototype is now released for Windows and OSX.
This prototype proposes an ergonomic mapping model to take advantage of the the full hand dexterity for expressive performance animation.

Try and evaluate

You can try the model and contribute with your evaluation by recording the 12 tests that are available. 8 tests with the puppet Luduvica and 4 tests with Cardboard boy puppet. If agree in making this test, just write a name in the label and start by pressing the “1” key or the “>” button. After the first test you can switch to the next by pressing the “>” button or the numeric keys “2”, “3”, and so on until the “8”. Then, you can jump to the next puppet by pressing the “2” button or the “Shift-F2” key. There are more 4 tests with this puppet that can be accessed with the “1”,”2”,”3”,”4” or using the “>” button. After finishing all tests you can send all the files that were saved to “virtual.marionette@grifu.com”. You can find or reproduce the file by pressing “browse”. The files are text files and motion capture binary files.

How to use

Just use your hand to drive the puppets. This prototype presents 4 distinct puppets with different interaction approaches that you can experiment by pressing 1,2,3,4 buttons.

• The palm of the hand drives the puppets position and orientation
• The little finger drives the eye direction (pupil) with 2 DOF (left and right)
• The ring finger and middle finger are mapped to the left and right eye brows with 1 DOF (up and down)
• The Index finger drives the eyelids with 1 DOF (open and close)
• The thumb finger is mapped to the mouth with 1 DOF (open and close)

There are other interaction methods with each puppet:

1. Luduvica – a digital glove puppet. This yellow bird offers a simple manipulation by driving just the puppet””””””””””””””””s head. If you use a second hand, you can have two puppets for a dramatic play. The performance test (7 key) allows you to drive the puppet in the Z-deph.
2. Cardboard Boy (paperboy) – a digital wireless marionette (or rod puppet). This physics-based puppet extends the interaction of Luduvica by introducing physics and a second hand for direct manipulation. As in traditional puppetry, you can play with the puppet with an external hand.
3. Mr. Brown – is a digital marionette/rod. This physics-based puppet is a similar approach to the paperboy but with more expressions. A remote console interface allows to drive a full set of facial expressions and the second hand allows the digital puppeteer to drive the puppet””””””””””””””””s hands (although not available in this version)
4. Minster Monster – is a digital muppet with arm and hand. Facial and hands are the most expressive components of our bodies as well in monster characters. While the head of the puppet is driven by the palm of the hand, the mouth is controlled by a pinch gesture in a similar way to traditional muppets. A second hand drives the arm and hand of the monster for expressive animation.

Live cameras and lighting

To simulate a live show you can switch among cameras, change the lighting setup or even focus and defocus the camera.

It also presents live camera switching and lighting control through remote OSC in port “7000” by using applications such as TouchOSC (try the“simple” preset)
.Switch between the 4 cameras with the messages “/2/push1” .. to “/2/push4”, to manipulate the lights use “/1/fader1” to “/1/fader4”, to focus and defocus the camera 3 on the Cardboard and Mr. Brown scenes use the “/1/fader5”. You can also navigate through scenes with the addresses “/2/push13” to “2/push16”.

Download

maniPULLaction for OSX

maniPULLaction for OSX - Digital Hand Puppetry with Leapmotion

1 file(s) 44.57 MB

Download

maniPULLaction for Windows

maniPULLaction for Windows - Digital Hand Puppetry with Leapmotion

1 file(s) 42.64 MB

Download

Info

Developed by Grifu in 2015 for the Digital Media PhD at FEUP (Austin | Portugal) supported by FCTIt requires the Leapmotion device.

Demo video

[vimeo 140126085 w=580&h=326]

Hellblade is a film in development that demonstrates the potential of combining High DOF digital puppetry with real-time top-quality rendering.

This is an amazing breakthrough demonstrating how digital puppetry methods (motion capture) can provide expressive animation which is rendered in real-time through CG top quality graphics.

Another step forward in the development of live films.

We reach a new era in the production of animation. Forget about off-line rendering, we are now living in real-time. This means that digital puppetry will soon become the hot topic. If we can render animation with film quality in real-time then, we need generate animation in real-time, and digital puppetry is our solution to achieve a brand new style of animation.

This short film was produced with Unity 5.4 beta version using the new cinematic sequencer tool.

It implements real time area lights and makes use of the physics simulation plugin CaronteFX.

Live cinema is arriving and with the recent release of Marza movie pipeline for Unity we are closer to this new realm.

Marza allows the production of movies inside the Unity platform changing the process of movie-making. Now, is more close to the concept of Machinima including digital puppetry methods to produce performance animation which is recorded to the film medium in real-time.

MARZA brings video game technology into the film production.

MARZA Movie Pipeline for Unity — Key features 

Pursuing rich animation expression

• Alembic importer co-developed with Unity Technologies
• Reproducing rich animation expression and advanced, complex VFX

Implementing modern texture expression

• Skin Shader: Character skin textures reproduction
• Eye Shader: Character eye expression
• Fake Fur Shader: Character “fur” expression

Dramatic improvement for efficient visual creation through automatic shot scene creation

• All required assets, camera and animations are automatically read by the Unity platform, allowing for instant visual checking
• Asset updates after scene creation

Enhancing visual quality
From Unity to Composite Software

• Using Frame Capture, co-developed with Unity Technologies
• Capturing game view as a sequence of OpenEXR image files
• Using render pass system to output additional information as extra OpenEXR layers

Thanks to the collaboration with Unity Technologies, we are happy to announce that some of the key features in the “MARZA Movie Pipeline for Unity” are already available in open source, under the MIT license at github, for Unity and the CG community at large.

• Frame Capture – https://github.com/unity3d-jp/FrameCapturer
• Alembic importer – https://github.com/unity3d-jp/AlembicImporter

Marionette Programming Language

Pull The Strings is a digital puppetry visual node-based control environment for performance animation.
It acts as a middle-ware interface between puppeteers and puppets, between device drivers and driven objects, between input interfaces and applications.

[vimeo 158084097 w=580&h=326]

It is a marionette programming engine that works as a digital mediator between devices and applications, it provides the building blocks for the digital puppeteer to establish the manipulation semantics. It is a visual programming language inspired by the strings of the marionettes and by the patch cords that connect modules in the old analog video synthesizers. In this environment the signals are processed and sent to the network. Finally the data arrives to Remote Control for Unity which is a plugin that facilitates the mapping of OSC messages.

It was designed with a minimalistic but intuitive interface and was developed in C++ with Openframeworks making use of multiple add-ons to provide their main features.
A multidisciplinary enviorment (MMMM: Multi-plataform, Multi-modal, Multi-mediator, Multi-user).
An enviorment for artists with non-programming background. Design for digital puppeteers.

Pull the strings is a middle-ware, an interface between applications, devices, and performing objects. It can be consider a visual programming environment for remote control made for artists and designers.
Sometimes you find hard to use one environment or application that offers all those nice features that your are looking for, so why not use a multiple set of applications in real-time ?
The goal of Pull the Strings is to facilitate the use and control of all your resources found in your computer and on the network.
An abstraction from technology making use of generic signals and communication protocols.
Connects, and transforms signals from input and virtual devices into performing object controls. Map and orchestrate multimedia data using OSC, DMX, MIDI and VRPN
Is a remote control application, an interface that connects inputs to outputs and provides a series of functionalities that helps to control the animation. It is a OSC-based environment and all nodes work with OSC format.

Version Alpha 0.050316 (first release 6 March 2016)
Pull The Strings will become open source soon. It is completly free to use in your art, research, or professional projects.
Built with [openFrameworks](http://www.openframeworks.cc).
It is developed by Luís Leite (GRIFU) for the Digital Media PhD with the support of FCT – Fundação para a Ciência e a Tecnologia.

## Download

Download Pull The Strings:

[Mac OS X 10.9 (32 bits) only ]

Pull The Strings Alpha 0.0316 (OSX-32bit)

1 file(s) 6.04 MB

Download

## Demo video

## License

Copyright 2012-2016 Luís Leite

GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.

Preamble

The GNU General Public License is a free, copyleft license for
software and other kinds of works.

## Pull The Strings Interface

Pull The Strings allows you to create, save and load your projects.
You should start by always saving your project first which will create a folder to gather all the files needed.
To navigate through the canvas use the key, to open the menu use the and then start writing the operator name or use the mouse to navigate.
To delete an operator just use the to access to operators options click the <*> on the operator.
Connect nodes from the outputs to the inputs. Add new input nodes by clicking the <+ O> button.
You can create multiple graph windows.

### OSC Communication
Pull The Strings is based on OSC protocol. It receives OSC messages and sends OSC messages and all its interface communicate internally via OSC.
You can send several messages from one output port.
It supports bonjour so you can easely find Pull The strings with Bonjour compatible applications, such as TouchOSC.
It makes use of RemoteUI to remote control the operators with other applications or devices.

### Pull The Strings operators

input.number (to create a Float or Int inside the interface or with remote applications)
input.string (to create strings)
math.scale (a scalar function with a learning input range option)
osc.combine (combine several OSC messages into one, it allows message syncincing)
osc.expand (expands an OSC message into seperated out nodes, it allows to inspect the data types and choose the address)
osc.port.in (opens port for incoming OSC data)
osc.port.out (opens port for sending)
print (for debugging, it prints in the interface the incoming messages)
show.plot (makes a plot from the values)

## Version History

– Alpha 003 (March 5th 2016)

## Support


## Other features
Pull The Strings as other operators that are not yet avaiable such as:
Timelines
LeapMotion support
Wiimote support
Math operations
Recording OSC performance

## Developed with
Pull The Strings is based on ofxDuct and makes use of many other ofxAddons such as:
ofxRemoteUI
ofxLeapMotion
Wiiuse
ofxBonjour
ofxAppGLFWindowMulti
ofxHistoryPlot
ofxUI
ofxTimecode
ofxMSATimer
ofxRange
ofxTimeline
ofxOsc
ofxGui
ofxMSAInteractiveObject