Magical VR at your fingertips!

Haptics invade The Magic Castle for Halloween

This is a reprint from an article posted by my partners and colleagues at UltraHaptics about an exciting project we created for Halloween at The Magic Castle:

The Magic Castle in Hollywood is one of the most exclusive clubhouses you will come across. Countless celebrities have visited and a who’s who of magicians have performed at the world-famous castle. Now for their best and biggest week of the year, Castle guests can experience a new type of magic with our mid-air haptics technology, as part of the Halloween Week special, INVASION!

With magic in every corner of the Magic Castle, how will you know what’s real and what’s a figment of your imagination?

MAGIC CASTLE HALLOWEEN INVASION

To celebrate All Hallow’s Eve, The Magic Castle has been transformed for the entire Halloween Week to handle this extraterrestrial alien assault. Deep below the century-old castle, the remaining humans (those who were not wiped out by the invasion) stage their last-ditch efforts to save the planet and repel the alien invaders.

To escape the aliens, valiant recruits are invited to join “The Resistance” and battle their way through a variety of exciting experiences and games. To power, the plasma cannons defending the Castle, guests in the Bunker must reach into the “Energy Chamber.” Using only their hands, they must gather invisible radioactive energy into balls of sizzling power. With our mid-air haptics solution, the guests actually see and feel the power as it builds, swirls and flows around their fingers. They then slide the growing balls of pulsating energy toward the various awaiting power conduits to fuel the cannons, which their friends can target to bring down the invading alien saucers.

The invasion is in full swing, and hundreds of guests each night are stepping up to take their turn defending the Castle and saving the planet.

All we can say is prepare for the unexpected…

As magicians, producers and imagineers we are always looking for new tools and devices to help realize new illusions, fantasies and dreams. Ultrahaptics is an exciting new tool in our arsenal. With its flexible haptic projection arrays, we can truly paint and animate the sensation of touch, much as we control and manipulate light and sound.

As a programmer, I am delighted to have the opportunity to work with the Ultrahaptics team. They have been a valuable addition to our project and we are only just beginning to discover the exciting possibilities of this new sensory dimension.

– Harry Evry, Producer, Programmer and Technical Director – Alien Invasion at The Magic Castle and Head of Education – VR/AR Association

TRICK OR TREAT?

Guests are not allowed to take photographs or video whilst inside, so evidence of this alien invasion is rare. But we’ve managed to get a behind-the-scenes look inside of Magic Castle and we’re bringing you with us…

Watch the Video!

https://youtu.be/yNHUKMwprhs

Life Lessons from a Self-Driving Car

How do you make your game or virtual world play-test itself?

Recently I built a TRON-inspired maze exploration game as part of an Escape Room project. In an effort to expedite the testing process, I decided to create an autonomous self-navigating demo component to repeatedly run the course.

Programming its paths procedurally would simply have the game engine repeat my own patterns and behavioral assumptions, so instead I borrowed some AI concepts from the latest autonomous vehicles.

It took about 6 hours to code a simple C# neural network from scratch, and then about two nights of automated training to make the virual self-driving ship passably good at playing the game. The training process was completely automated, and generally took place while I slept, or, at times, tried to sleep while gazing idly entranced at the flashing blinking TRON-like display.

There are several ways to train a neural nerwork. I could only evaluate the performance of this AI player by how well it actually played the game, so I used forward propagation (updating values in realtime while repeatedly playing the game), a reward system (the score) and applied a “genetic evolutionary algorithm”. (Bonus word, 4000 points on scrabble)

All of this is just a fancy way for saying that I let the neural-network try to play the game using 55 different randomly chosen sets of network parameters. The computer literally rolled the diced 92 times to make each of the 55 numeric patterns.

These sets of trial parameters ( technically called weights and biases) were then tested and sorted by the scores they achieved. Each round, the top 7 scoring sets, were then mated (randomly combined, giving priority to the most successful) to create new and hopefully even better sets of neural network weights and biases.

The first several rounds of trials weren’t very impressive. Many sets wouldn’t move at all, while others spun in circles like a top. If we got lucky we might see player rush wildly into a wall. The average score from the third round was about 22 points, but like watching a child taking first a step and then two, the robotic player began to get the hang of the game.

By the time it had tried the level a few thousand times, almost human-like skills began to emerge, but more importantly for my purposes, the game level itself had been tested more than a thousand times with a variety of random and drastically different paths, control patterns and input conditions.

The game never crashed or failed, but the neural-network did reveal a couple exploits or scoring logic errors which a shrewed user (and apparently a barely intelligent mindless matrix of randomly evolved numbers) could take advantage of.

As one example, the scoring system reflected the forward distance and speed the player traveled, the virtual AI discovered that it could quickly accumulate a very high score by racing forward then slowly rolling back, over and over again.

Exploit fixed! This is pretty impressive, when you consider that these exploits were discovered and logged while the programmer slept, and before a single human user ever actually alpha tested the game.

Another lesson that became obvious after the first night of training was that simply counting the obvious scoring points of the game wasn’t quite enough. Because the neural network was constantly learning and evolving based only on the score, it was necessary to adjust this score to account for and disincentivise certain specific undesirable patterns of action.

For instance, without any penalty adjustment, the neural network often chose to jump between jarring and abrupt joystick oscillations, virtually pounding the joystick one way and then the other throughout the game. This wasn’t a bad habit to test, but was surely annoyingly painful to watch. A couple lines of added code to monitor and deduct points for rapid abrupt joystick reversals, and like a repentant child the neural-network quickly began to model a much more elegant and mature pilot.

Instilling patterns and teaching desirable skills to a robotic neural network is much like teaching life’s lessons to a child, student, pet or employee. It seems somewhat wrong to group these all together, but like these real-world neural beings, AI neural networks respond to the obvious and those not so obvious systems of incentive, penalty, reward and punishment. The more clear, comprehensive and well considered the feedback, the more efficient and effective the training will be. The hardest challenge of programming the autonomous player wasn’t programming the neural network or the genetic learning algorithm. The biggest challenge that took a little experimentation was determining the right mix of incentives and penalties to properly motivate and encourage the robotic AI to master the game.

If you have never watched a neural-network interactively train, I highly recommend it. It can be slow and frustrating and at times somewhat comical, but much like seeing a child achieve a milestone or a pet learn a new trick, it can feel proudly rewarding when your AI robot brain takes even a small step closer to its life’s ambition of navigating a maze or taking over the world. If you have, please share your favorite exploits in AI behaviour and training.

If you enjoyed this entry, and especially if you didn’t, be sure to join me next week, as together we explore Life Lessons from a 2006 Hyundai Sonata. (Excuse the delay, as the Sonata issue has been indefinitely delayed due to unanticipated technical difficulties…)

VR Scripting Tutorials

With its large active community, an ever growing plug-in library, and almost universal platform crosscompatabiliyy, Unity is quickly becoming the leading platform for Augmented Reality and Vrirual Reality development. 

Unity scripts and components can be programmed in several ways, but it’s native language is C#.  For those new to programming, I have posted a couple introductory micro-pattern tutorials, which can be found on the new tutorials page of this site, revealed using the menu button above, 

These were developed as part of my introductory VR programming class, currently offered at StudioArts in Los Angeles.  Enjoy!

VR Mountains in Cyberspace

Anyone who visits a Disney park will notice that the longest lines tend to form around its mountains.  True these tend to be the roller coasters, but there is much more at here.  It is no coincidence that The Materhorn, Space Mountain, Thunder Mountain and Splash Mountain seem to gather crowds like lanterns gather bugs in a dark Florida swamp.

When Walt Disney was designing his great experiment in environmental design, Disneyland, he and his team quickly recognized that high peaks and towers draw visitors’ attention.  Disney described these architectural landmarks as serving like the small bits of hotdog stuck to the end of a long stick.  Studio animal wranglers used these on set to keep Lassie looking and walking in the desired direction.  Like Lassie’s favorite treats, Disney called these iconic landmarks “weenies” and at Walt Disney Imagineering the name has stuck.

Marty Sklar, Disney Legend and long-time imagineering chief, describes “weenies” as “visual magnets.”  Like magnets, they attract the eye and pull the guests along in the desired direction.  Where most other parks and resorts post wayfinding signs with big pointy arrows, Disney has never needed a sign to point anyone towards Sleeping Beauties Castle, Tomorrowland, Spaceship Earth, The Tower of Terror, The Haunted Mansion or Its a Small World.

In each of these examples, and so many more, the architecture itself acts as a sign, a beacon and even a preview of the adventure waiting within.  These “weenies” become iconic visions in the minds of the guests, memorable symbols of the promise of the epic experiences that await inside.  Few symbols in the history of the world are more recognized or beloved than Disneyland’s Sleeping Beauty’s Castle or Cinderella’s Castle at Walt Disney World.

When guiding VR travelers on their virtual adventures, Virtual Reality Designers may devise mountains, towering landmarks and other virtual “weenies” to focus guests attention and draw them towards new portals, sites and adventure.

– Harry Evry

Harry Evry is an author, reviewer, designer, producer, programmer, technical director, fx supervisor, composer, engineer and imagineer.  He has been a pioneer in digital production, post-production, interactive media, virtual reality, themed entertainment and video games.  His work has contributed to such respected companies and brands as Disney, Pixar, Universal, Warner Bros., Lorimar, Mattel, Paramount, Alf, Jurassic Park, Backdraft, WaterWorld, Max Steel, StarTrek and Indiana Jones.

Going with the Flow in Cyberspace

If virtual reality is to fulfill the promise of transporting us into another world, we must feel truly free to explore that world. 

In the VR Villlage at SigGraph 2016, Keio University and NTT Media Intelligence Laboratories invited guests to experience a very cool demonstraton.  In the aptly named VR Planet booth, VR travelers, with a viewer on their heads and two Vive controllers strapped to their shoes, stepped onto one of three illuminated spots highlighted on an empty floor.    

Like Mary Poppins stepping into chalk paintings on the pavement, the VR traveler found themselves in the middle of one of three different crowded and bustling cities. Once teleported into their chosen location, guests could turn their heads and look around the environments, but once they stepped away from the center of the scene, the scene would fade away.

This was a very creative and effective demonstration, but as cool as it was to “beam” into a cityscape, and it was very cool, the immersion felt a bit limited and incomplete, because I was, of course, frozen in one spot as throngs of people flowed all around me.  

Mobility and navigation will play an important part of many effective VR experiences.  Obviously the design, planning and production resources to enable this freedom will be significant, but without considering the specific methods of production, how will VR designers urge users to express their freedom of mobility freely, yet express it along the path and in the directions anticipated by the content creators.

One answer became apparent as a, perhaps unanticipated consequence of this great demonstration.  Magicians and amusement park operators have long understood that people tend to flock together and mirror the people around them.  Perhaps you have noticed at conventions or in theme parks that the booths or attractions with the biggest crowds or longest lines often draw crowds faster and faster.  Long lines tend to get longer and longer, even when there may be a much shorter line leading exactly the same place.

The same dynamic can apply in virtual reality. When immersed in the crowded street scenes of VR Planet, I felt compelled to follow the crowd.  I couldn’t follow them, but I did have total freedom to look around.  There too, however, I found myself turning with the crowd. Even as I consciously tried to catch the action in all directions, my attention continually drifted back to align with the direction of the people around me.  Magicians have long appreciated this particular phenomenon.

Like great magicians and motion picture directors, Virtual Reality Designers must become masters of attention direction, manipulating the users focus and driving their explorations subtly along their chosen paths.

– Harry Evry

Harry Evry is an author, reviewer, designer, producer, programmer, technical director, fx supervisor, composer, engineer and imagineer.  He has been a pioneer in digital production, post-production, interactive media, virtual reality, themed entertainment and video games.  His work has contributed to such respected companies and brands as Disney, Pixar, Universal, Warner Bros., Lorimar, Mattel, Paramount, Alf, Jurassic Park, Backdraft, WaterWorld, Max Steel, StarTrek and Indiana Jones

Video Games and Virtual Reality

Video Games and Virtual Reality

As a gamer, game designer and virtual reality developer, it seems natural to see the fields of video games and virtual reality as two sides of the same discipline.  Many new games and most game platforms are becoming more immersive.  Most virtual reality experiences are, in fact, powered by popular game engines like Unity3d and the Unreal Engine.  The lines between these industries are already fuzzy at best.  Some might argue that the two art forms are really one and the same.  Though the two industries are at least currently very different and distinct.

Can any video game be considered a virtual reality experience?  

According to some popular dictionaries they might.  Miriam-Webster offers the simple definition, or virtual reality, VR, as “an artificial world of images and sounds created by a computer that is affected by the actions of a person who is experiencing it”

The online dictionary site, Dictionary.com defines VR, as “a realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by movement of the body.”   A joystick or game-pad does rely on body movement, perhaps even more than most current VR headsets.  A Microsoft Kinect or Nintendo Wii game certainly meets this requirement. 

Oxford Dictionaries expand on this notion, defining VR as “the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors.”  This definition eliminates many arcade and console games, but still would include anything played with a Kinect or Wii type device.  

Ironically all of these definitions focus on the hardware interface by which a game is played.  Does this mean that any game played through a VR device should automatically be considered virtual reality?  Perhaps, some might argue that any act performed in front of a movie camera should be considered cinema.  

In 1902, it was.  The modern frontier of virtual reality isn’t very different today.    Trending “virtual reality” apps include things like virtual movie theaters, where users watch standard cinematic movies projected within their headsets onto a virtual flat movie screen.  Other popular “VR” apps place the user into an animated or recorded virtual environment with little or no story, little motion or no interaction.   Like the movies of the late 1800’s, these could be seen as legitimate stepping stones in the art of virtual reality, but if this is state-of-the-art virtual reality, then the game industry has long since left it in the dust, or has it?

As I wrote the last paragraph, I asked a young waitress what she would most like to experience in virtual reality.  She immediately told me she would want to go to an exotic location, travel or go site seeing.  Though not scientifically significant, this answer certainly does not fall within any common school of modern game design.  Game designers might propose that players want game play, adventure, challenge, goals, reward, agency, story and drama.  As a game designer, I would be very inclined to offer the players missions to accomplish, secrets to discover and obstacles to overcome.  None of these things would necessarily make for a better walk on the beach, view of Niagara Falls or tour of The Louvre.  

Surely the design of great virtual reality experiences must have its own rules, grammar and discipline.  Perhaps Virtual Reality Design might borrow more from exhibit, museum and theme park design practices.  It is reasonable to assume then that the design of games played in virtual reality would also follow somewhat different mechanics than the normal principles of game design suggest.  

What will be the break-out games and blockbuster genres of virtual reality?  How will designers harness the immersive power of VR to create the type of addictive amusements or epic experiences that will turn VR players into passionate residents of cyberspace? These are the questions on the forefront of virtual reality today and this is the mission of a very new emerging industry that many believe may someday surpass the combined output of both the motion picture and video game industries.   

– Harry Evry

Harry Evry is an author, reviewer, designer, programmer, technical director, fx supervisor and imagineer.  He has been a pioneer in digital production, post-production, interactive media, virtual reality, themed entertainment and video games.  His work has contributed to such respected companies and brands as Disney, Pixar, Universal Studios, Warner Bros., Lorimar, Mattel, Paramount Pictures, Alf, Jurassic Park, Backdraft, WaterWorld, The Magic Castle, Max Steel, StarTrek and Indiana Jones.