Tag Archives: Google Lens

Find the hidden stories behind art at the de Young with Google Lens

One of the privileges of working at the de Young museum in San Francisco is getting to regularly spend time in front of masterworks by artists like Ruth Asawa, Albert Bierstadt, and Mary Cassatt, and learn about the often fascinating stories surrounding their art. Spanning four centuries, the de Young museum’s American art collection includes paintings, sculpture, and decorative art from the 17th century to the present day. We have so many stories to tell.

As the museum’s director of digital strategy, it’s my job to find ways to make these stories more readily accessible for our visitors and to help people understand what the art says about the world, and the cultures, viewpoints, and moments in time that don’t always fit within the short labels in the galleries.

Our newest collaboration with Google Arts & Culture shows visitors the hidden stories behind the paintings in this collection. Now, using Google Lens, you can search what you see. Point your phone's camera at a work like Edmund Charles Tarbell’s The Blue Veil, and you’ll have a curator at the tap of your finger to tell you learn more about the artist’s origins, and his fascination with the veil.

Find out more with Google Lens

Learn more about art with Google Arts & Culture and Google Lens.

This is a way for artists to share their perspective, too. In a new exhibition, Detour, artist Ana Prvački takes you on a tour of the museum, guiding you to specific spots and asking you to rethink parts of the museum visitors many not normally consider, such as the material of the museum’s copper facade. Visitors can trigger Prvački’s short videos on mobile devices via Google Lens at sites throughout the free public spaces of the museum. When you watch the videos, it feels like you’re getting a personal tour from the artist herself.

If you can’t make it to San Francisco before the exhibition concludes in September, you can experience a version of Detour online on Google Arts & Culture.  

In a new exhibition, Detour, artist Ana Prvački takes you on a tour of the museum.

At I/O ’19: Building a more helpful Google for everyone

Today, we welcomed thousands of people to I/O, our annual developer’s conference. It’s one of my favorite events of the year because it gives us a chance to show how we’re bringing Google’s mission to life through new technological breakthroughs and products.

Our mission to make information universally accessible and useful hasn’t changed over the past 21 years, but our approach has evolved over time. Google is no longer a company that just helps you find answers. Today, Google products also help you get stuff done, whether it’s finding the right words with Smart Compose in Gmail, or the fastest way home with Maps.

Simply put, our vision is to build a more helpful Google for everyone, no matter who you are, where you live, or what you’re hoping to accomplish. When we say helpful, we mean giving you the tools to increase your knowledge, success, health, and happiness. I’m excited to share some of the products and features we announced today that are bringing us closer to that goal.

Helping you get better answers to your questions

People turn to Google to ask billions of questions every day. But there’s still more we can do to help you find the information you need. Today, we announced that we’ll bring the popular Full Coverage feature from Google News to Search. Using machine learning, we’ll identify different points of a story—from a timeline of events to the key people involved—and surface a breadth of content including articles, tweets and even podcasts.

Sometimes the best way to understand new information is to see it. New features in Google Search and Google Lens use the camera, computer vision and augmented reality (AR) to provide visual answers to visual questions. And now we’re bringing AR directly into Search. If you’re searching for new shoes online, you can see shoes up close from different angles and even see how they go with your current wardrobe. You can also use Google Lens to get more information about what you’re seeing in the real world. So if you’re at a restaurant and point your camera at the menu, Google Lens will highlight which dishes are popular and show you pictures and reviews from people who have been there before. In GoogleGo, a search app for first-time smartphone users, Google Lens will read out loud the words you see, helping the millions of adults around the world who struggle to read everyday things like street signs or ATM instructions.

Google Lens: Urmila’s Story

Google Lens: Urmila’s Story

Helping to make your day easier

Last year at I/O we introduced our Duplex technology, which can make a restaurant reservation through the Google Assistant by placing a phone call on your behalf. Now, we’re expanding Duplex beyond voice to help you get things done on the web. To start, we’re focusing on two specific tasks: booking rental cars and movie tickets. Using “Duplex on the Web,” the Assistant will automatically enter information, navigate a booking flow, and complete a purchase on your behalf. And with massive advances in deep learning, it’s now possible to bring much more accurate speech and natural language understanding to mobile devices—enabling the Google Assistant to work faster for you.

We continue to believe that the biggest breakthroughs happen at the intersection of AI, software and hardware, and today we announced two Made by Google products: the new Pixel 3a (and 3a XL), and the Google Nest Hub Max. With Pixel 3a, we’re giving people the same features they love on more affordable hardware. Google Nest Hub Max brings the helpfulness of the Assistant to any room in your house, and much more.

Building for everyone

Building a more helpful Google is important, but it’s equally important to us that we are doing this for everyone. From our earliest days, Search has worked the same, whether you’re a professor at Stanford or a student in rural Indonesia. We extend this approach to developing technology responsibly, securely, and in a way that benefits all.

This is especially important in the development of AI. Through a new research approach called TCAV—or testing with concept activation vectors—we’re working to address bias in machine learning and make models more interpretable. For example, TCAV could reveal if a model trained to detect images of “doctors” mistakenly assumed that being male was an important characteristic of being a doctor because there were more images of male doctors in the training data. We’ve open-sourced TCAV so everyone can make their AI systems fairer and more interpretable, and we’ll be releasing more tools and open datasets soon.

Another way we’re building responsibly for everyone is by ensuring that our products are safe and private. We’re making a set of privacy improvements so that people have clear choices around their data. Google Account, which provides a single view of your privacy control settings, will now be easily accessible in more products with one tap. Incognito mode is coming to Maps, which means you can search and navigate without linking this activity with your Google account, and new auto-delete controls let you choose how long to save your data. We’re also making several security improvements on Android Q, and we’re building the protection of a security key right into the phone for two-step verification.

As we look ahead, we’re challenging the notion that products need more data to be more helpful. A new technique called federated learning allows us to train AI models and make products smarter without raw data ever leaving your device. With federated learning, Gboard can learn new words like “zoodles” or “Targaryen” after thousands of people start using them, without us knowing what you’re typing. In the future, AI advancements will provide even more ways to make products more helpful with less data.

Building for everyone also means ensuring that everyone can access and enjoy our products, including people with disabilities. Today we introduced several products with new tools and accessibility features, including Live Caption, which can caption a conversation in a video, a podcast or one that’s happening in your home. In the future, Live Relay and Euphonia will help people who have trouble communicating verbally, whether because of a speech disorder or hearing loss.

Project Euphonia: Helping everyone be better understood

Project Euphonia: Helping everyone be better understood

Developing products for people with disabilities often leads to advances that improve products for all of our users. This is exactly what we mean when we say we want to build a more helpful Google for everyone. We also want to empower other organizations who are using technology to improve people’s lives. Today, we recognized the winners of the Google AI Impact Challenge, 20 organizations using AI to solve the world’s biggest problems—from creating better air quality monitoring systems to speeding up emergency responses.

Our vision to build a more helpful Google for everyone can’t be realized without our amazing global developer community. Together, we’re working to give everyone the tools to increase their knowledge, success, health and happiness. There’s a lot happening, so make sure to keep up with all the I/O-related news.

Source: Android


Helpful new visual features in Search and Lens

Sometimes, the easiest way to wrap your head around new information is to see it. Today at I/O, we announced features in Google Search and Google Lens that use the camera, computer vision and augmented reality (AR) to overlay information and content onto your physical surroundings -- to help you get things done throughout your day.

AR in Google Search

With new AR features in Search rolling out later this month, you can view and interact with 3D objects right from Search and place them directly into your own space, giving you a sense of scale and detail. For example, it’s one thing to read that a great white shark can be 18 feet long. It’s another to see it up close in relation to the things around you. So when you search for select animals, you’ll get an option right in the Knowledge Panel to view them in 3D and AR.

ar-search-shark.gif

Bring the great white shark from Search to your own surroundings.

We’re also working with partners like NASA, New Balance, Samsung, Target, Visible Body, Volvo, Wayfair and more to surface their own content in Search. So whether you’re studying human anatomy in school or shopping for a pair of sneakers, you’ll be able to interact with 3D models and put them into the real world, right from Search.

ar-search-visible-body.gif

Search for “muscle flexion” and see an animated model from Visible Body.

New features in Google Lens

People have already asked Google Lens more than a billion questions about things they see. Lens taps into machine learning (ML), computer vision and tens of billions of facts in the Knowledge Graph to answer these questions. Now, we’re evolving Lens to provide more visual answers to visual questions.

Say you’re at a restaurant, figuring out what to order. Lens can automatically highlight which dishes are popular--right on the physical menu. When you tap on a dish, you can see what it actually looks like and what people are saying about it, thanks to photos and reviews from Google Maps.

Consumer_Menu.gif

Google Lens helps you decide what to order

To pull this off, Lens first has to identify all the dishes on the menu, looking for things like the font, style, size and color to differentiate dishes from descriptions. Next, it matches the dish names with the relevant photos and reviews for that restaurant in Google Maps.

Consumer_TrainTicket.gif

Google Lens translates the text and puts it right on top of the original words

Lens can be particularly helpful when you’re in an unfamiliar place and you don’t know the language. Now, you can point your camera at text and Lens will automatically detect the language and overlay the translation right on top of the original words, in more than 100 languages.

We're also working on other ways to connect helpful digital information to things in the physical world. For example, at the de Young Museum in San Francisco, you can use Lens to see hidden stories about the paintings, directly from the museum’s curators beginning next month. Or if you see a dish you’d like to cook in an upcoming issue of Bon Appetit magazine, you’ll be able to point your camera at a recipe and have the page come to life and show you exactly how to make it.

Consumer_BonAppetit.gif

See a recipe in Bon Appetit come to life with Google Lens

Bringing Lens to Google Go

More than 800 million adults worldwide struggle to read things like bus schedules or bank forms. So we asked ourselves: “What if we used the camera to help people who struggle with reading?”

When you point your camera at text, Lens can now read it out loud to you. It highlights the words as they are spoken, so you can follow along and understand the full context of what you see. You can also tap on a specific word to search for it and learn its definition. This feature is launching first in Google Go, our Search app for first-time smartphone users. Lens in Google Go is just over 100KB and works on phones that cost less than $50.

Consumer_LensGo.gif

All these features in Google Search and Google Lens provide visual information to help you explore the world and get things done throughout your day by putting information and answers where they are most helpful—right on the world in front of you.

Source: Search


The era of the camera: Google Lens, one year in

THE CAMERA: IT’S NOT JUST FOR SELFIES & SUNSETS

My most recent camera roll runs the gamut from the sublime to the mundane:  

There is, of course, the vacation beach pic, the kid’s winter recital, and the one--or ten--obligatory goofy selfie(s). But there’s also the book that caught my eye at a friend’s place, the screenshot of an insightful tweet and the tracking number on a package.

As our phones go everywhere with us, and storage becomes cheaper, we’re taking more photos of more types of things. We’re of course capturing sunsets and selfies, but people say 10 to 15 percent of the pictures being taken are of practical things like receipts and shopping lists.

To me, using our cameras to help us with our day-to-day activities makes sense at a fundamental human level. We are visual beings—by some estimates, 30 percent of the neurons in the cortex of our brain are for vision. Every waking moment, we rely on our vision to make sense of our surroundings, remember all sorts of information, and explore the world around us.  

The way we use our cameras is not the only thing that’s changing: the tech behind our  cameras is evolving too. As hardware, software, and AI continue to advance, I believe the camera will go well beyond taking photos—it will help you search what you see, browse the world around you, and get things done.

That’s why we started Google Lens last year as a first step in this journey. Last week, we launched a redesigned Lens experience across Android and iOS, and brought it to iOS usersvia the Google app.

I’ve spent the last decade leading teams that build products which use AI to help people in their daily lives, through Search, Assistant and now Google Lens. I see the camera opening up a whole new set of opportunities for information discovery and assistance. Here are just a few that we’re addressing with Lens:

GOOGLE LENS: SEARCH WHAT YOU SEE

Some things are really hard to describe with words. How would you describe the dog below if you wanted to know its breed? My son suggested, “Super cute, tan fur, with a white patch.” 🙄

shibu

With Lens, your camera can do the work for you, turning what you see into your search query.

shibu lens gif

So how does Lens turn the pixels in your camera stream into a card describing a Shiba Inu?

The answer, as you may have guessed, is machine learning and computer vision. But a machine learning algorithm is only as good as the data that it learns from. That’s why Lens leverages the hundreds of millions of queries in Image Search for “Shibu Inu” along with the thousands of images that are returned for each one to provide the basis for training its algorithms.

shibu search

Next, Lens uses TensorFlow—Google’s open source machine learning framework—to connect the dog images you see above to the words “Shibu Inu” and “dog.”

Finally, we connect those labels to Google's Knowledge Graph, with its tens of billions of facts on everything from pop stars to puppy breeds. This helps us understand that a Shiba Inu is a breed of dog.

Of course, Lens doesn’t always get it right:

lens gourd

Why does this happen? Oftentimes, what we see in our day-to-day lives looks fairly different than the images on the web used to train computer vision models. We point our cameras from different angles, at various locations, and under different types of lighting. And the subjects of these photos don’t always stay still. Neither do their photographers. This trips Lens up.

We’re starting to address this by training the algorithms with more pictures that look like they were taken with smartphone cameras.

This is just one of the many hard computer science problems we will need to solve. Just like with speech recognition, we’re starting small, but pushing on fundamental research and investing in richer training data.

TEACHING THE CAMERA TO READ

As we just saw, sometimes the things we're interested in are hard to put into words. But there are other times when words are precisely the thing we’re interested in. We want to look up a dish we see on a menu, save an inspirational quote we see written on a wall, or remember a phone number. What if you could easily copy and paste text like this from the real world to your phone?

To make this possible, we’ve given Lens the ability to read and let you take action with the words you see. For example, you can point your phone at a business card and add it to your contacts, or copy ingredients from a recipe and paste them into your shopping list.

lens recipe

To teach Lens to read, we developed an optical character recognition (OCR) engine and combined that with our understanding of language from search and the Knowledge Graph. We train the machine learning algorithms using different characters, languages and fonts, drawing on sources like Google Books scans.

Sometimes, it’s hard to distinguish between similar looking characters like the letter “o” and zero. To do this, Lens uses language and spell-correction models from Google Search to better understand what a character or word most likely is—just like how Search knows to correct “bannana” to “banana,” Lens can guess “c00kie” is likely meant to be “cookie”—unless you're a l33t h4ck3r from the 90s, of course.

We use this OCR engine for other uses too like reading product labels. Lens can now identify over 1 billion products from our Google Shopping catalog—four times the number it covered at launch.

THE CAMERA AS A TOOL FOR CURIOSITY

When you’re looking to identify a cute puppy or save a recipe, you know what you want to search for or do. But sometimes we're not after answers or actions—we're looking for ideas, like shoes or jewelry in a certain style.

Now, style is even harder to put into words. That’s why we think the camera—a visual input—can be powerful here.

With the style search feature in Lens you can point your camera at outfits and home decor to get suggestions of items that are stylistically similar. So, for example, if you see a lamp you like at a friend’s place, Lens can show you similar designs, along with useful information like product reviews.

lens lamp

THE ERA OF THE CAMERA

A decade ago, I started at Google as a bright-eyed product manager enamored with the potential of visual search. But the tech simply wasn’t there. Fast forward to today and things are starting to change. Machine learning and computational photography techniques allow the Pixel 3 to capture great photos both day and night. Deep learning algorithms show promise in detecting signs of diabetic retinopathy from retinal photographs. Computer vision is now starting to let our devices understand the world and the things in it far more accurately.

Looking ahead, I believe that we are entering a new phase of computing: an era of the camera, if you will. It’s all coming together at once—a breathtaking pace of progress in AI and machine learning; cheaper and more powerful hardware thanks to the scale of mobile phones; and billions of people using their cameras to bookmark life’s moments, big and small.

As computers start to see like we do, the camera will become a powerful and intuitive interface to the world around us; an AI viewfinder that puts the answers right where the questions are—overlaying directions right on the streets we’re walking down, highlighting the products we’re looking for on store shelves, or instantly translating any word in front of us in a foreign city. We’ll be able to pay our bills, feed our parking meters, and learn more about virtually anything around us, simply by pointing the camera.

In short, the camera can give us all superhuman vision.

Step into the Sights of Sound with Pixel 3

To show you some of the features on the new Pixel 3 phone, we created Sights of Sound powered by YouTube Music, an immersive pop-up experience coming to San Francisco, Los Angeles, Austin, Miami and Atlanta this fall.

Visitors step inside rooms inspired by four musicians and use a Pixel 3 to navigate, capture and share the experience. To bring each room to life, you’ll need to solve a puzzle using Google Lens, the visual tool that lets you search what you see. Throughout the pop-up, you and your friends can try out Pixel’s brilliant camera by using Group Selfie Cam, Top Shot and Photobooth Mode.

We collaborated with Ella Mai, Future, J Balvin and Marshmello to create each immersive space in the pop-up. Dive deep into Ella Mai’s Inner Reflections, sit like royalty among snakes and nine-foot tall golden lions in Future’s Trap Mansion, dance your way through J Balvin’s Reggaeton World, bounce your troubles away in Marshmello’s Happier Place and capture it all on the Pixel 3’s incredible camera.

For more information on dates, times and locations for Sights of Sound, visit g.co/sightsofsound. If you’re in San Francisco, Los Angeles, Austin, Miami or Atlanta, we hope to see you there!

See your world differently with Playground and Google Lens on Pixel 3

Our phone’s camera has become an important part of our lives. We use it to capture moments that matter to us, like memories with our families, and the small things we don’t want to forget, like handwritten notes or parking tickets. And now, thanks to advancements in computer vision, our camera can understand our world and help in new ways.

Today we introduced our most helpful camera yet on Pixel 3 and Pixel 3 XL, so you can turn your world into a playground and do more with what you see.

Introducing Playground, a new way to create and play

From reenacting scenes in “Star Wars: The Last Jedi” to hanging out with Eleven from “Stranger Things,” you’re already having fun with AR Stickers. Today, we’re taking a big step forward by introducing Playground, a new mode in the Pixel camera to create and play with the world around you. It helps you bring more of your imagination to a scene with cameos from your favorite superheroes, stickers that animate around you and fun captions that put words where the action is.

Playground brings you more powerful AR experiences and uses AI to recommend content for expressing yourself in the moment. You can make your photos and videos come to life with Playmoji—characters that react to each other and to you—and tell a richer story by adding fun captions or animated stickers you’ll recognize from Gboard.

Comp-1.gif

You can snap photos on the back camera (left) or take selfies with Playmoji that respond to your facial expressions (right).

Playground also works in selfie mode, so now you can up your selfie game by standing next to characters you love, like Iron Man from the Marvel Cinematic Universe. New packs for Weather, Pets, Sports and Signs let you have everyday fun, and coming later this year, you'll be able to sharpen your dance skills with moves from Childish Gambino.

Iron Man _ Blog%2FSocial (1).gif

Marvel Studios’ Avengers

And for those moments when you aren’t quite sure where to start, Playground makes real-time suggestions to recommend content based on the scene you’re in. Are you walking your dog? Cooking in the kitchen? Gardening in the backyard? Playground uses advanced computer vision and machine learning to recommend relevant Playmoji, stickers and captions to populate the scene.

AR_Playground_Garden.gif

Scene suggestions

Once you’ve got your perfect shot, sharing with Playground is easy—with just a few taps straight from the camera.

Do more with what you see with Google Lens

Last year with Pixel 2, we introduced Google Lens to help you do more with what you see, whether its finding an outfit you like, copying and searching text, or identifying that cute dog in the park. This year, we’ve integrated Lens even more into Pixel 3 for a faster, more helpful experience.

LensKeynote_Menu_1.gif

Lens Suggestions

Lens is already a great way to quickly call a number, visit a URL, scan a QR code, and add an email address to your contacts. On Pixel 3, we’re introducing Lens Suggestions to bring these common actions right into the main Pixel camera. Simply point your camera, and with a single tap, call or save a phone number on a takeout menu, send an email right from a flyer, or open an address in Google Maps. This is all done on-device and takes advantage of Pixel Visual Core, so it doesn’t require an internet connection.

We’ve also made it even easier to get to the full Lens experience on Pixel 3—just long press in the camera. It’s a fast and convenient new way to access Lens.

Lens in Overview.gif

Lens from recent apps screen on Pixel 3

Finally, we’re making Lens accessible from the recent apps screen on Pixel 3, so regardless of the app you’re in, Lens can help. Just drag up from the home button and long press on an image. It’s great for when you see an outfit you like while browsing the web or scrolling through an app.

With Google Lens and Playground, Pixel 3’s camera lets you do more with what you see and have fun with the world around you. Google Lens is available on all Pixel devices, with Playground on Pixel 3 first and coming to Pixel 1 and 2 soon. Preorder Pixel 3 or learn more on the Google Store.