Tag Archives: India

Google Home and Google Home Mini launches in India

Bringing together the best of Google’s AI, software and hardware, now with a desi twist

Whether you’re getting the kids ready for school, doing a batch of laundry, or answering the doorbell for the morning vendors, Indian homes are busy ones. From catching that Bollywood blockbuster on your smart TV, to whipping up a quick Chole Bhature, to sinking into soulful Sufi tunes at the end of a tiring day, you can now get hands-free help.

Beginning today, Indian users can welcome in their lives Google Home -- our voice-activated speaker powered by the Google Assistant. With a simple “Ok, Google” or “Hey Google”, you can get answers, turn up the music, manage everyday tasks or even control smart devices around your home.

Google Home understands Indian accents, and will respond to you with uniquely Indian contexts. What’s great about the Google Assistant is that it’s the same across all your devices, so that it works seamlessly for you wherever you need a helping hand. You can for instance ask it for the quickest route to office, then tell it to push the directions to Google Maps on your smartphone, and you’re ready to navigate as you head out.

Designed to fit seamlessly into your home
We didn’t want Google Home to feel like a gadget, and took inspiration from consumer products that are commonly found in homes, like wine glasses, candles, and even donuts for Mini.

The top surface has LEDs that provide visual feedback when Google Home recognizes “Hey Google”, so you know when it is listening. In those rare moments when voice won’t do, the top surface is also a capacitive touch panel. You can simply use your finger to pause the music or adjust the volume.
Google Home was designed with two microphones to enable accurate far-field voice recognition. The microphone system uses a technique called neural beam forming. We’ve simulated hundreds of thousands of noisy environments and applied machine learning to recognize patterns that allow us to filter and separate speech from noise. This allows us to deliver best-in-class voice recognition and minimize error rates -- even from across the room. Home will be available in India in the Chalk color variant.
Google Home Mini is sleek and smooth, with no corners or edges. And it's small enough to easily place anywhere in your home. It’s almost entirely enclosed in custom fabric. We created this material from scratch, right down to the yarn. It’s durable and soft, but also transparent enough to let through both light and sound. And it is available in Chalk and Charcoal, with Coral coming soon. The four LED lights underneath the fabric that light up to show you when it hears you or when it’s thinking. Mini has far-field mics so it can hear you even when there’s music playing or loud noise in the background: its circular design it can project 360-degree sound, with just one speaker.

These devices join the Made by Google family of hardware products in India, and will be available for purchase online exclusively on Flipkart, and in over 750 retail stores across the country including Reliance Digital, Croma, Bajaj Electronics, Vijay Sales, Sangeetha, and Poorvika.

Tap into the power of Google with your Assistant
Need answers to a problem? Ask questions, translate phrases, run simple maths calculations and look up the meaning of a word. Too busy to stay on top of the news? Ask and you shall receive the latest stories from sources such as Times of India, NDTV, Dainik Bhaskar, India Today, Aaj Tak and more. Need a helping hand in the kitchen? Find ingredient substitutes, pull up nutritional information and unit conversions without having to wash your atta-covered fingers.

Google Home is truly ‘desi’
With a distinctly Indian voice, your Assistant on Google Home speaks and understands your language. Ask it “Hey Google, how desi are you?”, put its cricket knowledge to the test with “Hey Google, what is a silly point?”, tell it to “Play songs from the movie Satte Pe Satta”, or even get step-by-step cooking instructions in the kitchen with, “Hey Google, give me a recipe for Dum Biryani”.

Get personalised help for your everyday tasks
The Google Assistant on Google Home has been designed to help you get more stuff done when you have your hands full. With your permission, it will help with things like your commute, your daily schedule and more. And the best part? Up to six people can connect their account to one Google Home, so if you ask your Assistant to tell you about your day, it can distinguish your voice from other people in your family, and give you personalised answers. Just ask “Hey Google, tell me about my day” or say, “Hey Google, how long will it take to get to work?” and you’ll get up to speed on everything you need to know. It can wake you up in the morning (or let you snooze), set a timer while you’re baking, and so more.

Turn up those tunes
Find the right rhythm for every occasion, whether you’re getting into the zone with sunrise yoga, hosting a dinner party, or burning off calories dancing with your little ones. You can play songs, playlists, artists, and albums from your favourite music subscription services like Google Play Music (with a six-month subscription, on us), along with offers from Saavn and Gaana*. You can also pair Google Home or Home Mini with your favorite Bluetooth speaker and set it to be the default output for all your music.

Control your smart home
Google Home can help you keep track of everything going on in your home--you can control your lights, switches and more, using compatible smart devices from brands like Philips Hue, D-Link and TP-Link. Just ask your Google Home, and your Assistant will turn off the kitchen light. If you have a Chromecast, you can also use voice commands to play Netflix, or YouTube on your TV and binge watch your favorite shows. Enjoy multi-room by grouping Google Home devices together (with Chromecast Audio, Chromecast built-in and Bluetooth speakers) to listen to the same song in every room.

A speaker for any occasion
Whether you’re hosting a dinner or a solo dance party, Google Home delivers crystal-clear sound and creates an enjoyable listening experience. Plus, we designed Home to fit stylishly into any room. And you have the option to customize the base with different colors to reflect your home’s style.

With Google Home, we’re working with our partners to bring you many great launch offers: when buying Google Home or Google Home Mini on Flipkart you get a free JioFi router along with special offers on exchange and streaming music subscriptions; when buying a Google Home at Reliance Digital or MyJio stores you get a free JioFi router with 100GB of high-speed 4G data (worth Rs 2,499)**, and at select Philips Hue and Croma outlets you get a Philips Hue + Google Home Mini at a special bundled price. Also ACT Fibernet retail customers subscribing to 12-month advance rental plans of 90MBPS and above, will receive a Google Home Mini. And above all, users get 10 percent cashback when purchasing using HDFC Bank credit cards***.

Google Home and Google Home Mini will be priced at Rs 9,999, and Rs 4,499 respectively.

It’s just the beginning...
Your Assistant on Google Home will continue to get better over time as we add more features (look out for Hindi support coming later this year!) And Google Home is open to third-party apps for the Assistant, so expect even more of your favourite services and content.

Posted by Rishi Chandra, VP,  Product Management, Google Home

*Both available from April 10 to October 31, 2018, for all Google Home and Home Mini users in India
**Offer valid until 30th April 2018
***Cashback limited to 10% of MRP

Fostering the Artificial Intelligence and Machine Learning ecosystem in India

Artificial intelligence has the potential to improve people’s lives in profound ways — from helping diagnose diseases and breaking down language barriers to making businesses more efficient. And we’re just at the beginning of what’s possible. We believe that ultimately, AI will help tackle huge challenges like healthcare, environmental protection and other social and developmental problems, while also spurring innovation for businesses and developers.

The opportunity is huge and not constrained by location – a company in  Bangalore or Gurgaon could serve the whole world. In fact, a recent report by Accenture concluded that India, by embracing AI technologies could add nearly $1 trillion to its GDP by 2035.  
India already has some of the key ingredients to become a major force in leading the next generation of disruptive innovation in machine learning (ML): a tech-savvy talent pool, renowned universities, healthy levels of entrepreneurship and strong corporations. This does however require for the whole ecosystem, be it government, industry professionals, academia or the developer community to come together and identify areas and opportunities;  participate and contribute to high quality research and innovation; and turn these systems into effective business models.

We at Google want to actively work with the community to foster and nurture the ecosystem. As a first step towards that we organised a workshop today at our campus in Bangalore, to bring together a cross section of the AI and ML community in India.
Jeff Dean, Senior Fellow, Google and Prabhakar Raghavan, VP, Apps Google Cloud kicked off the workshop by sharing Google’s vision for AI and the work we are doing to help businesses and developers innovate with AI. Faculty and researchers from the IITs and other leading universities, and industry practitioners from both startups and bigger companies including Amazon, Flipkart, LinkedIn, Myntra, Microsoft and Ola participated and spoke on the ongoing research and work being undertaken in India in many areas of AI/ML such as Deep Learning, Computer Vision, Natural Language Processing, ML Systems, and Generative Models.

Bringing together people from different parts of the community in a workshop is one step we’re taking to inspire people in India to innovate with AI. We also provide researchers, companies and developers with tools to solve complex problems with machine learning, including the open-source machine learning framework TensorFlow, custom-built machine learning chips TPUs, and Cloud AI––a suite of products to help businesses build their own machine learning powered services from pre-trained APIs to Cloud AutoML and the Cloud Machine Learning Engine. And just a couple of weeks ago, we made a set of educational resources about machine learning and AI available, including a free online machine learning crash course that anyone can use to learn and practice machine learning concepts.

Every year, Google gives large grants to over 250 academic research projects world-wide, supports PhD students, and hosts thousands of interns. At the event in Bangalore today, we announced an additional focused award round to support five faculty members in India on their AI research proposal. This is over and above the grants we have already made since 2012 to faculty and PhD students in India to support academic research and they are given as unrestricted grants with no return obligation (IP etc) to Google.

Likewise for entrepreneurs, we are working on an accelerator program, based in India, focused primarily on AI/ML technologies.  Our global accelerator program has already supported over 30 Indian startups of which six are focused on applied AI/ML innovation.

When every developer, entrepreneur or researcher can leverage the power of intelligent systems, there’s no limit to what AI/ML can accomplish, and that benefits all of us.

Posted by Pankaj Gupta and Anand Rangarajan, Engineering Directors at Google India

Say namaste to your Hindi Google Assistant!

Everybody needs a helping hand sometimes -- that’s why we built the Google Assistant, a way to have a conversation with Google that helps you get things done. Over the last year, we’ve launched the Assistant in Hindi in Allo, brought the English Assistant to more phones, and made a special version available for Reliance Jio feature phones. Now, more Indian users can get help from their Hindi Assistant -- starting today, the Assistant in Hindi is rolling out to Android phones 6.0+ (Marshmallow and above) and soon coming to Android 5.0 Lollipop and iPhone devices.

The Google Assistant lets you have a conversation with Google to help you get things done in your world -- from telling you about your day, to finding the fastest route to work, or just setting an alarm for the next day -- all in Hindi. To try it out, just touch and hold the Home button or say “Ok Google” on eligible smartphones and your personal Google Assistant will be ready to help you throughout your day. Even when your hands are full, you can quickly send text messages, set reminders, or get directions with the Google Assistant.
Powered by machine learning, the Google Assistant is built on two decades of experience in Search as well as natural language understanding, computer vision, and understanding user context. That’s how your Assistant is able to understand intent behind words to handle follow-up questions and complex, multi-step tasks. And it'll get better over time -- with your permission, it can learn your preferences, your likes and your dislikes -- all done in a private, secure way that puts you in control.

The Google Assistant is truly Indian, it’s your helpful dost that speaks our language and understands the things you care about, from finding biryani recipes, to pulling up the latest cricket score, or finding directions to the nearest ATM. In order to make it even more useful to Indians over time, developers and businesses can now build Actions for the Hindi Assistant through the developer platform Actions on Google. Once an action is built, you can just say “Ok Google, talk to…” and access the service or content straight through your Google Assistant.

Here are some examples of questions you can ask your Google Assistant:
“Sabse kareeb Punjabi restaurant kahaan hain?“
“Dadar tak pahunchne mein kitna samay lagega?“
“Cricket ka score kya hain?“
You can even instruct it…
“Kal subah mujhe saat baje jagaao“
“Selfie kheencho“
“Daddy ko SMS bhejo "5 minutes mein pahunchenge"

For an overview of all Actions already available for the Hindi Assistant, you can check out this site.

With today's update, the Google Assistant in Hindi is now available on Android phones 6.0+ (Marshmallow and above), in Allo and soon on Android 5.0+ Lollipop, Android Oreo (Go edition), and iPhones (iOS 9.1 and above). To access the Hindi Google Assistant, set your device language to Hindi and update your Google Search app to the latest version. For more information please see the Google Assistant website.

Posted by Purvi Shah, Technical Program Manager, Assistant

Searching for Indian addresses on Google Maps gets better; Plus Codes — a simple location-based digital addressing system; Voice Navigation in six additional Indian languages

At Google Maps, our objective has always been about organizing all location-related information, and making it more accessible and relevant for Indian users. Address search is critical for delivering on this mission.

In India, we know how challenging it can be to reach a given residential address. They are unique in format, and vary across regions, localities, and use cases. While some addresses are well-defined by street names and house numbers that are easy to find, others can be long-winded and hard to locate. The other reality is that millions of people and places in India are hard to locate -- especially those in remote areas. We are deeply committed to helping find solutions to these challenges.

Introducing Plus Codes
We have developed an open sourced solution -- Plus Codes --  which represents a simple and consistent addressing system that works across India and globally. This system is based on dividing the geographical surface of the Earth into tiny ‘tiled areas’, attributing a unique code to each of them. This code simply comprises a ‘6-character + City’ format that can be generated, shared and searched by anyone -- all that’s needed is Google Maps on a smartphone. The open source nature of Plus Codes means that applications that uses location services can easily incorporate it on their platforms for free.
Creating and using Plus Codes

To use a Plus Code, simply enter it into the Search field on Google or Google Maps, mobile or desktop. That’s it -- you’ll be instantly shown the location!

Plus Codes can be used for a wide variety of reasons including communicating the venue of a temporary event, guiding emergency services to afflicted locations, and providing an identifiable location for complicated addresses.

For more information on Plus Codes, you can visit plus.codes.

Easily add a missing address to Google Maps
In another step aimed at facilitating accurate and easy searching on Maps, we are now introducing ‘Add an Address’ -- a feature that enables you, our users, to contribute to the Maps experience from the Google Maps app. Similar to adding businesses, users can submit new or missing addresses through this feature, and we'll make sure the address is searchable in due course after verification. And yes, you do get Local Guides points for each valid submission!

Adding a missing address

Understanding addresses, especially in the Indian context (being so unique and varied,) has always been a relentless pursuit for us. After extensive research, starting today, we are making it simpler to search for addresses in India, with Smart Address Search. This is an innovative approach to providing the best estimate of the location of an address: when people aren’t aware of an exact address, they just don’t give up but try to reference other information such as a nearby landmark, business or a locality to get closer to the final destination. Smart Address Search helps doing exactly that: if Google Maps doesn’t understand the address precisely, it will try to use all the pieces of information in the address and provide options of landmarks and points of interest that the user can best identify with -- exactly like a person would have!
Using smart address search

From finding to getting there -- now in six additional languages
Since the time we launched Navigation in Hindi three years ago, we have consistently received requests to be able to change the voice-guided navigation from English to a local language. Today, we are happy to bring voice navigation in six additional Indian languages: Bengali, Gujarati, Kannada, Telugu, Tamil, and Malayalam.
Choosing Indian languages for voice navigation

It doesn’t stop there -- India continues to inspire us, and pushes us to reimagine what Maps can do for people. And we are just getting started as we continue to build features and expand our partnerships to make the Google Maps experience more comprehensive, accurate, and reliable for our users.

Go ahead, explore your world!

Posted by Suren Ruhela, Director, Google Maps Next Billion Users

With just a flick of a wand, “Harry Potter: A History of Magic” is on Google Arts & Culture

For Harry Potter’s fellow students at Hogwarts, “A History of Magic” is historian Bathilda Bagshot’s legendary chronicle of Wizarding history. And last year, we mere Muggles got our own version. “Harry Potter: A History of Magic” is an exhibition from the British Library containing rare books, manuscripts and magical objects from the British Library’s collection, capturing the traditions of folklore and magic from across the world, which are at the heart of the Harry Potter stories.

Turns out, the exhibition was more popular than the Three Broomsticks on a cold day … it quickly sold out. To bring the Harry Potter magic to more fans around the world, hundreds of the exhibition’s treasures from London as well as 15 online exhibits are now available in six languages (English, Spanish, French, German, Hindi and Brazilian Portuguese, and more coming soon) on Google Arts & Culture.

These examples shed light on what you’ll see in the exhibit. Lumos!

Jim Kay’s original painting of Harry Potter and Draco Malfoy playing Quidditch.
Still waiting for your Hogwarts letter? For now, you can take your own Herbology class with Culpepper’s Herbal (used by J.K. Rowling for inspiration for Herbology coursework).

J.K. Rowling’s original sketch of the Hogwarts’ grounds (with her meticulous notes included) and the original synopsis of the first Harry Potter book.

Explore the The Ripley Scroll, which describes how to make the fabled Philosopher’s Stone.

Check out the Divination room in 360-degree tour.

Julian Harrison, lead curator of the exhibition, shares “Ten Strange Things You Didn't Know About the History of Magic.
In a fascinating Q&A, Jim Kay (the famous illustrator of the books) what it was like to draw “The Boy Who Lived.”
Simon Costin, Director of Museum of Witchcraft and Wizardry talks about the “The Bewitching Collection.”
Bloomsbury Children’s Books publishing director Rebecca McNally talks about the “Harry Potter Effect” and the enduring magic of the series.
Visitors to the library explored the British Library’s 400-year old Celestial Globe (with constellations like Draco, Lupus and Sirius). Check out this video to see what it was like.
The British Library exhibit has proven that “interest in magic is a real global phenomenon, and has fascinated people for thousands of years,” says Julian Harrison, Lead Curator for Medieval Historical Manuscripts and “Harry Potter: A History of Magic.”
“The British Library is thrilled that our blockbuster ‘Harry Potter: A History of Magic’ exhibition can now be viewed on Google Arts & Culture. We’ve used medieval manuscripts, precious printed books and Chinese oracle bones to explore magical traditions, from the making of potions to the harvesting of poisonous plants, and from the study of the night sky to the uses of unicorns.”
To explore these magical traditions for yourself, check out The British Library collections online with Google Arts & Culture and on our iOS and Android apps

Posted by Suhair Khan, Program Manager Google Arts & Culture

Search for health symptoms in English and Hindi in the Google App

Picture this: you woke up with a headache. It’s been getting worse all day, and you aren’t sure if you should be worried or not. If you’re like many of us then chances are you’ll go online and search for your symptoms. You’re not alone. Roughly 1 percent of searches on Google are symptom-related. But health content on the web can be difficult to navigate, and tends to lead people from mild symptoms to scary and unlikely conditions, which can cause unnecessary anxiety and stress.

In 2016, we introduced health condition cards to make it easier for Indians to find reliable health information.  Starting this week, we will be adding information about commonly searched symptoms. Say you’re not feeling well and search for symptoms like “cough and pain”, we’ll show you a list of related conditions (“common cold, acute bronchitis, flu, pneumonia, chest infection”).  For individual symptoms like “सिरदर्द ” we’ll also give you an overview description along with information on self-treatment options and what might warrant a doctor’s visit. By doing this, our goal is to help you to navigate and explore health conditions related to your symptoms, and quickly get to the point where you can talk to a health professional or do more in-depth research on the web.

To make sure the information is tailored to India, we’ve been working closely with a team of medical doctors at Apollo Hospitals. “Apollo Hospitals has always been committed to empowering individuals with knowledge and involving them in their own care,” says Ms. Sangita Reddy, Joint Managing Director, Apollo Hospitals Enterprise Limited. “This is a significant trend and we are happy to have partnered on this initiative with Google. At Apollo Hospitals, we have always made optimal use of digital technology for the benefit of patients. The abundant experience and expertise of our consultants was drawn towards providing clinical validation of the symptoms to spread the message of awareness and prevention of diseases. With the launch of the Symptom Search Project, we aim to provide quality healthcare information which can be accessed by a billion Indians.

These search results are intended for informational purposes only, and you should always consult a doctor for medical advice and treatment. We hope that this basic information helps make it easier for you to know what questions to ask your doctor. So the next time you’re worried about an upset stomach, or want to look up symptoms for someone in your family, a Google search will be a helpful place to start.

The new symptom search experience will be available in Hindi and English on the Google App (Android and iOS).

Posted by Henrique Bejgel, Software Engineer, Google Search

Updates to YouTube Live Streaming

Together, we've experienced the biggest music, sports, science, culture and gaming events unfold live on YouTube. Now we’re introducing more ways to watch live videos and interact with your community in real time.

Catching up on the latest

Live chat plays a key role in creating connections between creators and their community. Today we are starting to roll out chat replay to YouTube, so you can follow the conversation even after a live stream is over. Live chat replays will show up alongside the video, exactly as it appeared live.

Making live streams more accessible

We launched automatic captions back in 2009, and since then, we’ve auto-captioned a staggering 1 billion videos. We’re now bringing English automatic captions to live streams.

When professionally provided captions aren’t available, our new live automatic captions provide creators a quick and inexpensive way to make live streams accessible to more people. With our live automatic speech recognition (LASR) technology, you’ll get captions with error rates and latency approaching industry standards. We'll roll this out in the coming weeks, and will continue to improve accuracy and latency of automatic captions.

More fun features for live streamers

Creators can now add a location tag to their mobile live streams and video uploads and share all their favorite hot spots with viewers. You can explore other videos with the same location tag by simply clicking on it. You can also use the location filter on the search results page to find other videos from a specific spot.
With live streams, you’ve found more intimate and spontaneous ways to share your thoughts, lives, and creativity. Take these features for a spin and show us your world!

Posted by Kurt Wilms, Live Video Product Lead, recently watched Falcon Heavy Test Flight

Google announces a ‘Developer Students Club’ summit, to nurture India’s next-generation in technology


Continuing to broaden the access to world class technology curricula and giving the large Indian student community a head start on latest technologies like Machine Learning, Artificial Intelligence and Mobile & Web Development, we are delighted to kick off the inaugural Developer Student Club Summit in Goa. The program is aimed at inspiring and training student ambassadors who will encourage students in their campuses to learn newer areas of technologies that will equip them with the right skill sets as they step out of colleges to join the workforce.

In the inaugural batch, 200 student ambassadors from 98 cities across 170 colleges in India, will attend the three day Summit (Feb 23- Feb 25) in Goa and learn about various emerging technologies like Mobile and Web development, Machine Learning, AR/VR, Artificial Intelligence, and Cloud Platforms. The summit will see a series of engaging sessions from experts from Google and industry, hands-on one-to-one mentoring activities, that will help the students to unlock their creative potential. The summit will also have a ‘Design Thinking’ workshop by the  University Innovation Fellows from across India, encouraging them to find creative and innovative solutions, using advanced technologies to solve for India’s complex problems. 

"It is an awesome feeling teaching others what you know; we are not only teaching but also growing ourselves. " -- Spoorthi V, Vidya Vikas Institute of Engineering and Technology, Mysuru

"I have special respect for DSC than other any other program previously I've been a part of because the DSC program is changing me into a entirely new person, a more mature, sensible and empathetic man and I really owe the DSC program a lot." -- Christy Anoop, CMR Institute of Technology, Bengaluru

This effort is inline with our larger program of training  two million developers in India.  Last year, we also announced 130K scholarships for both working professionals and students community, that gives them access to new-age technology and an opportunity to gain skills they require to be successful in the changing technology landscape. Our goal is to help nurture future developers and leaders and create a pool of highly skilled tech workforce, that is readily employable by the Industry and help spur innovation. 

We are excited about this new program and will host a series of such summits over the course of next 12 months. 

Posted by William Florance, Developer Products Group and Skilling Lead, Google

Google launches support for Tamil languages ads


After Hindi and Bengali, AdWords and Adsense to now support Tamil

India has 234* million Indian language users who are online (compared to 175 million English web users) and it is expected that another 300 million Indian language users will come online in the next three years. The most important aspect of making the web more useful and meaningful for Indian users is to make India’s Internet more representative of today’s India.  

To make the web more inclusive, useful and meaningful for Indians, It’s essential to work towards creating useful content in local languages and focus on native language advertising to establish a connection with the audience.

We have been working on this front for the last couple of years and we are delighted to announce Tamil support in our advertising products - Google AdWords and Google AdSense. Advertisers will now be able to develop a rich and seamless campaigns to target their audience through Tamil-language search and display ads. In addition to that, content creators who have websites and blogs in Tamil will be able to sign up for Google Adsense and run ads on their content and attract advertisers from across the globe.

We saw a huge acceleration in Hindi content creation, when we launched support for Hindi languages, and we hope that the addition of new languages like Bengali (launched late last year) and now Tamil will provide the necessary boost to content creation in local languages and grow digital advertising in Indian languages.

To start monetizing your Tamil content website with Google AdSense:
                      1. Check the AdSense program policies and make sure your website is compliant.
                      2. Sign up for an AdSense account
                      3. Add the AdSense code to start displaying relevant ads to your
*Source: Indian Languages - Defining India’s Internet Report by KPMG
By Shalini Girish, Director - Google Marketing Solutions, Google India

Assessing Cardiovascular Risk Factors with Computer Vision

Heart attacks, strokes and other cardiovascular (CV) diseases continue to be among the top public health issues. Assessing this risk is critical first step toward reducing the likelihood that a patient suffers a CV event in the future. To do this assessment, doctors take into account a variety of risk factors — some genetic (like age and sex), some with lifestyle components (like smoking and blood pressure). While most of these factors can be obtained by simply asking the patient, others factors, like cholesterol, require a blood draw. Doctors also take into account whether or not a patient has another disease, such as diabetes, which is associated with significantly increased risk of CV events.

Recently, we’ve seen many examples [1–4] of how deep learning techniques can help to increase the accuracy of diagnoses for medical imaging, especially for diabetic eye disease. In “Prediction of Cardiovascular Risk Factors from Retinal Fundus Photographs via Deep Learning,” published in Nature Biomedical Engineering, we show that in addition to detecting eye disease, images of the eye can very accurately predict other indicators of CV health. This discovery is particularly exciting because it suggests we might discover even more ways to diagnose health issues from retinal images.

Using deep learning algorithms trained on data from 284,335 patients, we were able to predict CV risk factors from retinal images with surprisingly high accuracy for patients from two independent datasets of 12,026 and 999 patients. For example, our algorithm could distinguish the retinal images of a smoker from that of a non-smoker 71 per cent of the time. In addition, while doctors can typically distinguish between the retinal images of patients with severe high blood pressure and normal patients, our algorithm could go further to predict the systolic blood pressure within 11 mmHg on average for patients overall, including those with and without high blood pressure.

LEFT: image of the back of the eye showing the macula (dark spot in the middle), optic disc (bright spot at the right), and blood vessels (dark red lines arcing out from the bright spot on the right). RIGHT: retinal image in gray, with the pixels used by the deep learning algorithm to make predictions about the blood pressure highlighted in shades of green (heatmap). We found that each CV risk factor prediction uses a distinct pattern, such as blood vessels for blood pressure, and optic disc for other predictions.

In addition to predicting the various risk factors (age, gender, smoking, blood pressure, etc) from retinal images, our algorithm was fairly accurate at predicting the risk of a CV event directly. Our algorithm used the entire image to quantify the association between the image and the risk of heart attack or stroke. Given the retinal image of one patient who (up to 5 years) later experienced a major CV event (such as a heart attack) and the image of another patient who did not, our algorithm could pick out the patient who had the CV event 70 per cent of the time. This performance approaches the accuracy of other CV risk calculators that require a blood draw to measure cholesterol.

More importantly, we opened the “black box” by using attention techniques to look at how the algorithm was making its prediction. These techniques allow us to generate a heatmap that shows which pixels were the most important for a predicting a specific CV risk factor. For example, the algorithm paid more attention to blood vessels for making predictions about blood pressure, as shown in the image above. Explaining how the algorithm is making its prediction gives doctor more confidence in the algorithm itself. In addition, this technique could help generate hypotheses for future scientific investigations into CV risk and the retina.

At the broadest level, we are excited about this work because it may represent a new method of scientific discovery. Traditionally, medical discoveries are often made through a sophisticated form of guess and test — making hypotheses from observations and then designing and running experiments to test the hypotheses. However, with medical images, observing and quantifying associations can be difficult because of the wide variety of features, patterns, colors, values and shapes that are present in real images. Our approach uses deep learning to draw connections between changes in the human anatomy and disease, akin to how doctors learn to associate signs and symptoms with the diagnosis of a new disease. This could help scientists generate more targeted hypotheses and drive a wide range of future research.

With these promising results, a lot of scientific work remains. Our dataset had many images labeled with smoking status, systolic blood pressure, age, gender and other variables, but it only had a few hundred examples of CV events. We look forward to developing and testing our algorithm on larger and more comprehensive datasets. To make this useful for patients, we will be seeking to understand the effects of interventions such as lifestyle changes or medications on our risk predictions and we will be generating new hypotheses and theories to test.

[3] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature (2017). doi:10.1038/nature21056

Posted by Lily Peng MD PhD, Product Manager, Google Brain Team