I grew up in Nantong, a beautiful, small coastal city in China, where the Yangtze River flows into the East China Sea. One of my earliest childhood memories was when my parents, during the long winter nights, would entertain me by narrating stories using hand shadows as puppets.
Later, I would see my first traditional shadow puppetry performance during a Lunar New Year family trip to Wuzhen. Over the years as I made my way around the world—whether in Indonesia, Egypt, or Greece—I found a form of shadow puppetry in local cultures, beautifully combining legends and traditions, music and art, imagination and craftsmanship. And it always made me think about those childhood nights with my family, and about passing down stories, connection, joy and love.
With technology, I'm hoping to help connect people to this ancient art form. In September last year, we built an interactive installation that used AI to help people explore shadow puppetry. Though it’s an ancient art, people connected with shadow puppetry in a new way, and after the conference, we decided to bring it online so that everyone could play. So today, we’re making it available as a new AI experiment, Shadow Art.
Shadow Art is a web browser-based game that lets you experience AI and shadow puppetry in a playful way. To bring what used to be an offline experience online, we used TensorFlow.js, a TensorFlow library which makes it easy to build and train a machine learning model directly in the browser.
How does Shadow Art work?
You use your hands to form one of twelve zodiac animals from the lunar cycle in front of your laptop or phone camera, trying to match your hand to the diagram on the screen. The “shadow” of your hands on the screen then transform into a shadow puppet animal. Sounds easy, right? Here’s the catch: we turned it into an interactive game where you have twenty seconds to form each animal. The goal is to go through the full lunar cycle as fast as possible.
The new experiment is now available in eleven language varieties, including English, Chinese, Thai, Bahasa Indonesia, Malay, Japanese, Korean, Spanish and Portuguese. In several countries around the world, our annual Lunar New Year Doodle is also celebrating the ancient storytelling art of shadow puppetry.
It’s been great to see the shadow puppets of my childhood come to life in Shadow Art. I hope you’ll have as much fun with it as we did (my personal record for the full Zodiac cycle? 2:23 mins ;)). Happy Lunar New Year to all of you!
