Disney's vision for augmenting creativity

By Will Freeman,


It should come as no surprise to anyone that Disney as a collective of creative minds understands the craft of entertaining.

But thanks to Disney’s prominence in the realm of mainstream, mass-market entertainment, however, it can be easy to miss the iconic studio’s efforts in the realm of R&D.

The Disney Research initiative sees several networked laboratories across the world take an academic approach to the pursuit of scientific and technological innovation. Much of the effort at those labs is to fuel Disney itself with new means by which to entertain, but the research teams are not precious with their work, sharing a great wealth of their findings and ideas.

The projects developed at Disney Research are wildly varied, from the captivating physical quilt embroidery game Threadsteading, which uses a hacked sewing machine, to the now famous BB-8 movie prop concept, which met the public as the Star Wars: The Force Awakens droid of the same name.


Considering that appetite for innovation, Disney labs have also long been looking at augmented reality, and ways to push the concept into the mainstream. And when it comes to AR, Disney Research doesn’t only see a means of entertainment; augmented reality is being fleshed out to serve as a tool that stimulates creativity. It’s a concept that could not only benefit the consumer, but also creators looking for new ways to deliver content that can thrive.

Mattia Ryffel is a research engineer at Disney Research Zurich, and one of the outfit’s leading proponents of its  ‘augmented creativity’ concept.

The idea is simple. Creative play has been important to founding the human experience for generations. In other words, as youngsters, we learn about the world through interaction founded in play.

“Creative play is an important aspect of children’s development, and kids need to tinker with things and explore; that active discovery helps us to achieve a memorable childhood, and gives us a lot of problem solving skills,” Ryffel offers, explaining the fundamental concept of what is sometimes termed ‘experimental learning’.

Interaction with one’s real-world surroundings is key, Ryffel asserts, be it in the form of listening, coloring, touching, exploring, or any of the other childhood mainstays that help us better understand the world as we mature.

“However, a lot of pre-packaged content – and on digital devices especially – has become more and more popular,” Ryffel continues. “So children spend a lot of time absorbed in digital worlds, consuming content. Very often this content doesn’t offer much in terms of creativity or an educational value.”

For anybody involved in the games industry since the 1990s, ‘educational gaming' may send a shudder down their spine. After all, the concept was once – unfairly or otherwise – synonymous with lower quality products.

Today, however, Disney is working on several innovative ‘augmented creativity’ AR projects that encourage experimental learning, while at the same time delivering creative worthy content that could also garner much commercial success.

Many of these projects provide a point of inspiration for developers keen to embrace AR.


Disney Research's AR colouring book app
The AR colouring book created at the Swiss Disney Research lab demonstrates the potential of 'scripting by colouring', where a real-world crayon could influence game character behaviors


Take Disney Research’s AR coloring books, for example, which see a child fill in a traditional 2D line drawing of a character. As the youthful user scribbles away with a crayon, that character also comes to life in AR as a 3D model, and is shaded in tandem with its two-dimensional cousin.

“The app lets kids express their creativity by drawing, and therefore using their imagination,” states Ryffel. “It enhances their sense of connection to characters. It motivates kids to colour more and more, and [it lets them] explore their work in 3D. This provides some educational value. It also opens up a lot of opportunities in augmented reality gaming.”

Ryffle also sees the potential of supporting something akin to ‘user painted content’ in future titles, or even ‘scripting by coloring’, where a player’s real-world shading could dictate the likes of character behaviors in-game.

Disney Research Zurich has also created a prototype music app built around marker cards illustrated with different instruments and musical styles. Scatter those cards on a table, and through the lens of AR users will find themselves arranging musical tracks, dropping in and out instruments, with captivating effect.

“This application has strong creative and educational aspects,” Ryffel suggests. “You can recreate a favorite song in a lot of different ways, or imagine those songs in other styles. You can combine and play with the instruments. This also teaches kids about music styles, how a song can be rearranged in different styles, and with different instruments.”

Disney Research's creative AR music app
Proving the continuing relevance of marker-based AR – and boasting great potential for multiplayer creative augmented reality – Disney Research's music app provides and orchestra in a pack of cards


Clearly possessing a mind full of ideas, Ryffel goes on to propose possibilities involving huge multiplayer orchestras creating music through AR, sharing yet another inspiring example with his fellow developers.

Ryffle is also besotted by the idea of ‘pervasive AR games’ played within and with real physical spaces.

“We have done a lot of experimenting with pervasive gaming in general,” he confirms. "The thing is, interacting components in computer games are very often limited to the screen, mouse, keyboard or gamepad, so you sit in front of your TV and – well – just sit there. You are removed from physical interaction with the world or other places.”

Inspired by those limitations of traditional forms of interaction, Disney Research Zurich crafted a multiplayer team game, based around defending yourself and friends from waves of aliens emerging from a real cube placed in the centre of a physical space. Importantly, it is a concept that can only be truly experienced with collaboration, and a little jostling of shoulders, as players gather and hustle around the six-sided centerpiece.

“This game makes you find strategies to cooperate and coordinate with your team,” says Ryffel. “It closes the gap between virtual and physical, real interactions. It teaches cooperation and communication, and, of course, it assures that you practice your communication skills.”

Elsewhere in Disney’s research labs there are frameworks underway that will let developers overlay AR content across whole cities, and projects to teach coding involving low-cost real-world robots that, when viewed via AR, demonstrate how coding is impacting their behavior.


Disney Research's Thymio robot coding project
By adding AR support to the educational Thymio robot coding concept, Disney Research was able to deliver clearer, more meaningful learning

These are just some examples of realizing ‘augmented creativity’ by conceiving apps that ask their players to do more than simply play. Ryffel and his team have shaped content that stimulates creativity and educates, from ways as obvious as a coloring book, to games that make a city a canvas for players to share their imagination.

“They explore and develop the concept of augmented creativity in different ways, and through different ways of interaction,” says Ryffel of his team’s example AR projects. “But they have common points. They all enhance real-world activities. They cultivate creativity through AR, and especially through AR interactivity. They foster education on many different levels, and finally, they open further possibilities in the field of augmented reality gaming and augmented reality interaction, as well as [helping answer] many research questions.”

AR can benefit from Disney’s history of innovation and entertainment, and increase their influence in education through augmented creativity.

For readers keen to learn more, Disney Research's augmented creativity paper is available at the lab's own website.


Got it

We use cookies to ensure that we give you the best experience on our website.

Click here for more information.