Virtual Marionette

Rui Torres, Luis Aly, Luis Grifu performing at Conde Duque in Madrid (Retroescavadora)

Retroescavadora (Rui Torres, Luis Aly, Luis Grifu) performed with Untitled at Conde Duque in Madrid last 19th of April. The Untitled performance is an experimental multimodal multimedia live performance of the POEX arquive where moving image, sound and manipulation are inter-connected in a real-time dialog.

Rui Torres performing at Conde Duque in Madrid (Untitled performance from Retroescavadora – Rui Torres, Luis Aly, Luis Grifu)

Retroescavadora was invited to be a part of a Poetry recital seeking new languages behind the technologies “La palabra en las periferias de la tecnología”. This session gathered Kònic Thtr (España), Retroescavadora (Portugal), Jörg Piringer (Austria) y Bell Dome (Brasil/Alemania).

More information at http://condeduquemadrid.es/evento/la-palabra-las-periferias-la-tecnologia-ii-edicion/

Untitled Performance

During the last three days I had the privilege to attend to the marionette workshop by Tommy Luther at ACE Teatro Bolhão.

This was an  intensive workshop of three days (27 and 28 of February, and 1 March)  to understand how to endow life to a marionette applying the manipulation principles of the Handspring Puppet Company, which are known by its international success on the play War Horse.

Tommy Luther was the director of the horse puppet which is manipulated by three puppeteers like in the Bunraku. He also worked manipulating objects for CG animation such as in the film Gravity (2009).

We have learn that breathing is an important aspect to: endow emotion to the puppet; to underline dynamic changes; to help to communicate the physical and psychologic state of the puppet; to unite all the puppeteers to became uno.

Wel also learn that it is very important to keep the space between puppeteers, as well the famous sentence that “less is more”.

Finally, the looking direction contributes not only to guide the audience direction to the audience, but also to give intelligence to the puppet. Before the action takes place, the puppet thinks and he looks to subject.

Virtual Marionette puppet tools were used to develop several interactive installations for the animation exhibition Animar 12 at the cinematic gallery Solar in Vila do Conde. The exhibition was debuted in the 18th of February of 2017.

Faz bem Falar de Amor

An interactive installation that challenges the participants to interpret with virtual characters some of the scenes from the animation music clip developed by Jorge Ribeiro for the music “Faz bem falar de amor” from Quinta do Bill.

Puppit tool was adapted to drive two cartoon characters (a very strong lady and a skinny young man) using the body motion of the visitors with with one Microsoft Kinect. The virtual character skeletons differ from the human body proportions. Thus, there are distinct behaviors on each puppet that do not mirror exactly the participant movement. Although our intention was to present this challenge to the participant for him to adapt his body to the target puppet we help him a little bit. To solve this discrepancy, I used two skeletons. A human like skeleton is mapped directly to the performer”s body. Then, the virtual character skeleton is mapped to the human clone skeleton with an offset and scale function. In this way, it was possible to scale up and down the movement on specific joints from the clone skeleton and make the virtual character behave more natural and cartoonish. This application was developed using Unity, OpenNI and Blender.

É Preciso que eu Diminua

This animation music clip was developed by Pedro Serrazina for Samuel Úria. It describes the story of a character that suffers from a scale problem transcending the size of the buildings. The character tries to shrink placing the arms and legs near to is body. On the other hand there is the need to free from the strict boundaries and expand all the things around him. To provide the feeling of body expansion pushing the boundaries away from his body the visitor drives a silhouette as a shadow with a contour line around him that grows when the participant expands his body and shrinks when he reduce his size. The silhouette captured by a Microsoft Kinect is projected on a set of cubes that deform the body shape. This application was developed with Openframeworks with ofxKinect and ofxOpenCV.

Estilhaços

For this short animation film produced by Jorge Miguel Ribeiro that addresses the portuguese colonial war there was the intention to trigger segments of the film when the visitor place his body above a mine. Two segments of the film show two distinct perspectives from the war, one from a father that experienced the war and the other from his child that understood the war through the indirect report of his father. A webcam was used to capture the position of the visitor”s body and whenever his body enters the mine space it sends an OSC message with a trigger to a video player application. Both video trigger and video player applications were developed with Openframeworks.

Untitled Anarchive is an experimental multimodal multimedia live performance of the POEX arquive. The Digital Arquive of Portuguese Experimental Literature was performed live at Salão Brazil on the 4th of February by members of Retroescavadora (Rui Torres, Luís Aly, and Luís Grifu) and their invited guests (Bruno Ministro and Sandra Guerreiro).

This collaborative intervention explores new approaches to experimental literature mixing media and manipulation techniques, in a interactive environment that offers a common space for a dialog between the inter-actors.

A demo and poster of Mani-pull-action was shown at the Austin|Portugal Annual Conference at UNL Lisboa on May 23rd and 24th, 2016.

For this exhibition Mani-pull-action was shown inside a paper theatre or Kamishibai as a physical connection or metaphor for the art of puppetry. The participants were invited to manipulate different puppets with their hands and experiment digital puppetry.

Each puppet required a different level of dexterity simulating direct and indirect manipulation with physics and inverse kinematics approaches. The participants could also play with virtual cameras and lights using a touch device with a set of controls.

This was another contribution for disseminating digital puppetry among the community and try to attract more enthusiasts to this form of art.

LEAP IT – STRING is a simple application that allows you to share the hand motion data captured by Leap Motion through network via OSC protocol with the fowling features:

– Record the performance
– Replay a recorded performance
– Select joints for broadcast
– Convert a recorded binary performance into a text file
– Hide/Show the GUI
– Convert the scale of the hand motion data
– Switch Palm Orientation between Pitch, Yaw, Roll or Yaw, Pitch, Roll

This application is a “string” from the Virtual Marionette Interaction Model developed by Luís Grifu for a research on digital puppetry. This application is free to use.

LeapIT String is now available for Mac and Windows

LeapIT String for Mac (v0.6)

1 file(s) 17.78 MB

Download

LeapIT String for Windows (v0.6)

1 file(s) 12.54 MB

Download

Hellblade is a film in development that demonstrates the potential of combining High DOF digital puppetry with real-time top-quality rendering.

This is an amazing breakthrough demonstrating how digital puppetry methods (motion capture) can provide expressive animation which is rendered in real-time through CG top quality graphics.

Another step forward in the development of live films.

We reach a new era in the production of animation. Forget about off-line rendering, we are now living in real-time. This means that digital puppetry will soon become the hot topic. If we can render animation with film quality in real-time then, we need generate animation in real-time, and digital puppetry is our solution to achieve a brand new style of animation.

This short film was produced with Unity 5.4 beta version using the new cinematic sequencer tool.

It implements real time area lights and makes use of the physics simulation plugin CaronteFX.

Live cinema is arriving and with the recent release of Marza movie pipeline for Unity we are closer to this new realm.

Marza allows the production of movies inside the Unity platform changing the process of movie-making. Now, is more close to the concept of Machinima including digital puppetry methods to produce performance animation which is recorded to the film medium in real-time.

MARZA brings video game technology into the film production.

MARZA Movie Pipeline for Unity — Key features 

Pursuing rich animation expression

• Alembic importer co-developed with Unity Technologies
• Reproducing rich animation expression and advanced, complex VFX

Implementing modern texture expression

• Skin Shader: Character skin textures reproduction
• Eye Shader: Character eye expression
• Fake Fur Shader: Character “fur” expression

Dramatic improvement for efficient visual creation through automatic shot scene creation

• All required assets, camera and animations are automatically read by the Unity platform, allowing for instant visual checking
• Asset updates after scene creation

Enhancing visual quality
From Unity to Composite Software

• Using Frame Capture, co-developed with Unity Technologies
• Capturing game view as a sequence of OpenEXR image files
• Using render pass system to output additional information as extra OpenEXR layers

Thanks to the collaboration with Unity Technologies, we are happy to announce that some of the key features in the “MARZA Movie Pipeline for Unity” are already available in open source, under the MIT license at github, for Unity and the CG community at large.

• Frame Capture – https://github.com/unity3d-jp/FrameCapturer
• Alembic importer – https://github.com/unity3d-jp/AlembicImporter

Marionette Programming Language

Pull The Strings is a digital puppetry visual node-based control environment for performance animation.
It acts as a middle-ware interface between puppeteers and puppets, between device drivers and driven objects, between input interfaces and applications.

[vimeo 158084097 w=580&h=326]

It is a marionette programming engine that works as a digital mediator between devices and applications, it provides the building blocks for the digital puppeteer to establish the manipulation semantics. It is a visual programming language inspired by the strings of the marionettes and by the patch cords that connect modules in the old analog video synthesizers. In this environment the signals are processed and sent to the network. Finally the data arrives to Remote Control for Unity which is a plugin that facilitates the mapping of OSC messages.

It was designed with a minimalistic but intuitive interface and was developed in C++ with Openframeworks making use of multiple add-ons to provide their main features.
A multidisciplinary enviorment (MMMM: Multi-plataform, Multi-modal, Multi-mediator, Multi-user).
An enviorment for artists with non-programming background. Design for digital puppeteers.

Pull the strings is a middle-ware, an interface between applications, devices, and performing objects. It can be consider a visual programming environment for remote control made for artists and designers.
Sometimes you find hard to use one environment or application that offers all those nice features that your are looking for, so why not use a multiple set of applications in real-time ?
The goal of Pull the Strings is to facilitate the use and control of all your resources found in your computer and on the network.
An abstraction from technology making use of generic signals and communication protocols.
Connects, and transforms signals from input and virtual devices into performing object controls. Map and orchestrate multimedia data using OSC, DMX, MIDI and VRPN
Is a remote control application, an interface that connects inputs to outputs and provides a series of functionalities that helps to control the animation. It is a OSC-based environment and all nodes work with OSC format.

Version Alpha 0.050316 (first release 6 March 2016)
Pull The Strings will become open source soon. It is completly free to use in your art, research, or professional projects.
Built with [openFrameworks](http://www.openframeworks.cc).
It is developed by Luís Leite (GRIFU) for the Digital Media PhD with the support of FCT – Fundação para a Ciência e a Tecnologia.

## Download

Download Pull The Strings:

[Mac OS X 10.9 (32 bits) only ]

Pull The Strings Alpha 0.0316 (OSX-32bit)

1 file(s) 6.04 MB

Download

## Demo video

## License

Copyright 2012-2016 Luís Leite

GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.

Preamble

The GNU General Public License is a free, copyleft license for
software and other kinds of works.

## Pull The Strings Interface

Pull The Strings allows you to create, save and load your projects.
You should start by always saving your project first which will create a folder to gather all the files needed.
To navigate through the canvas use the key, to open the menu use the and then start writing the operator name or use the mouse to navigate.
To delete an operator just use the to access to operators options click the <*> on the operator.
Connect nodes from the outputs to the inputs. Add new input nodes by clicking the <+ O> button.
You can create multiple graph windows.

### OSC Communication
Pull The Strings is based on OSC protocol. It receives OSC messages and sends OSC messages and all its interface communicate internally via OSC.
You can send several messages from one output port.
It supports bonjour so you can easely find Pull The strings with Bonjour compatible applications, such as TouchOSC.
It makes use of RemoteUI to remote control the operators with other applications or devices.

### Pull The Strings operators

input.number (to create a Float or Int inside the interface or with remote applications)
input.string (to create strings)
math.scale (a scalar function with a learning input range option)
osc.combine (combine several OSC messages into one, it allows message syncincing)
osc.expand (expands an OSC message into seperated out nodes, it allows to inspect the data types and choose the address)
osc.port.in (opens port for incoming OSC data)
osc.port.out (opens port for sending)
print (for debugging, it prints in the interface the incoming messages)
show.plot (makes a plot from the values)

## Version History

– Alpha 003 (March 5th 2016)

## Support


## Other features
Pull The Strings as other operators that are not yet avaiable such as:
Timelines
LeapMotion support
Wiimote support
Math operations
Recording OSC performance

## Developed with
Pull The Strings is based on ofxDuct and makes use of many other ofxAddons such as:
ofxRemoteUI
ofxLeapMotion
Wiiuse
ofxBonjour
ofxAppGLFWindowMulti
ofxHistoryPlot
ofxUI
ofxTimecode
ofxMSATimer
ofxRange
ofxTimeline
ofxOsc
ofxGui
ofxMSAInteractiveObject