To be, or not to be--that is the question, and one that takes on a new dimension for Hamlet when he is transposed into a 3D virtual world.
To Be With Hamlet is a live theatre performance in Social Virtual Reality that is being developed by a coalition of artists, engineers and programmers across NYU’s global network.
Combining live theatre, motion capture and virtual reality in a way that has never been done before, the show will allow up to 15 audience members to walk around and share the same virtual experience with Hamlet as he confronts the ghost of his murdered father on the battlements of Elsinore Castle.
To bring this experience to the virtual stage, NYU researchers in Shanghai, Abu Dhabi and New York have combined forces to direct and capture the actors’ performances and transpose them into a virtual social environment.
On October 14, during the 2016 Global Cre8 Summit in Shenzhen, Guangdong Province, Christian Grewell, Adjunct Assistant Arts Professor at NYU Shanghai and the creative lead of the M3diate production team which is developing the project’s virtual platform, gave a preview of the Hamlet experience to an audience of 2,000 tech entrepreneurs and developers.
We caught up with Grewell to find out more about the technology - and its possible applications beyond the theater.
Q: To Be With Hamlet is described as a live theater performance in Virtual Reality that allows people to have an intimate experience with Hamlet. Can you tell us more about the audience experience?
To Be With Hamlet allows audience members that may be miles apart to meet up and experience a live theatre performance in a virtual environment. After donning a VR headset, each audience member will find themselves on set and be able to hear and see the play as it is being performed live, and talk with fellow audience members.
Hamlet himself is a photorealistic avatar of a live actor, created by the NYU Tandon team using Optitrack face and motion capture technology. The actor’s performance is further brought to life with the integration of our custom-built M3diate platform which creates a spatially accurate audio environment: for example, the distance to the actor as well as the virtual physical environment of the virtual stage will all affect how the audience member hears the sounds in the virtual world.
Q: What is your role in the project?
NYU Shanghai partnered with NYU Abu Dhabi’s M3diate research project to experiment with real life uses of Virtual Reality around transcending physical locations (e.g. for classes that would span globally). We decided to focus on the audio experience. In doing so, M3diate also developed methods for sending audio and motion capture data through existing internet bandwidth that could be received at high quality and low-latency, which is one of the key features of the M3diate platform.
Because the main focus of NYU Tandon’s To Be With Hamlet event was on motion capture and performance, it was a no-brainer to collaborate on this project as M3diate would be able to provide the technology to get To Be With Hamlet to the audiences around the world.
Q: What have been some of the challenges during the production process?
What is ground-breaking, and therefore challenging, is taking the concept of social VR and putting it into the context of a traditional social event: the theater. To date, this has never happened before and companies such as AltVR or ConVRge, while creating environments for social VR gatherings, have never had a formal event offering in such a fashion.
There is a lot involved here that makes this very, very hard to do. The first challenge, is representation of the human form and motion in virtual space. In each frame (many, many times a second), we need to solve kinematics equations of a virtual skeleton to determine all of the joint parameters that provide a desired position of the end-effector (e.g. the actor's pinky finger).
The next challenge is how to send all of this data over the internet to the connected clients. The amount of data is massive so we built our own implementation and compression system. This is hard to do.
Finally, In terms of staging the events, it's a lot of setup, and a lot of technology has to come together to make it all work. For example, in order to have lifelike motion and ‘cross the uncanny valley’ (where human avatars that appear almost, but not exactly, like real human beings elicit feelings of eeriness and revulsion) we need to use a full motion capture suit, which takes time to set up. We then have the challenges of setting up all of the specific VR rooms and ensuring they are all connecting to the various servers in Shanghai, Abu Dhabi or NYC - it works, but is definitely not plug-and-play.
Q: What future applications of this technology do you foresee beyond virtual theatre?
There are hundreds if not thousands of use-cases for the applications of VR! But in terms of what we're trying to do, we've built this platform for immersive collaboration and communication in virtual worlds. Our biggest question is what are the novel use cases. For example, the inspiration for the entire project actually came from my Chinese teacher. She said that some day, we would be having our Chinese lesson in VR. This stuck with me, and is one of the major reasons that we chose to pursue 3D spatialized sound as the first 'presence multiplier'. We have a lot of prototypes on the roadmap, including collaborative home design and architecture, collaborative production design in VR, and yes, even language learning.
Q: How well equipped is NYU Shanghai for this kind of research?
NYU Shanghai is in the heart of the eastern hemisphere's VR Silicon Valley. While NYC is quickly establishing itself as the VR capital in the west, NYU Shanghai can tap into the enormous VR community of both enthusiasts and professionals, not to mention major companies that are all establishing VR divisions within their organizations here in China.
The CRE8 conference organizers have just lent NYU Shanghai the Optitrack motion capture equipment that they purchased for the conference. This is currently being set up at NYU Shanghai so that our students will be able to use it to further their own projects in this field. We could even do our own local version of HamletVR - perhaps Chinese Opera!
Q: Where can I experience Hamlet in VR?
The Hamlet VR team has been accepted to a number of conferences around the world, where we intend to perform a scene from Hamlet live for those audiences. We will be at VR Days conference in Amsterdam on Nov 8. We will be setting up the VR studio at NYU Shanghai for the Spring and private demos can also be organised at NYU Tandon in NY.
